diff --git a/examples/offline_inference/image_to_video/README.md b/examples/offline_inference/image_to_video/README.md index 257d6a42e..68bddca06 100644 --- a/examples/offline_inference/image_to_video/README.md +++ b/examples/offline_inference/image_to_video/README.md @@ -1,8 +1,7 @@ # Image-To-Video -This example demonstrates how to generate videos from images (image-to-video) -using Wan2.2, LTX2, HunyuanVideo-1.5, and Cosmos3 with vLLM-Omni's offline -inference API. +This shared example generates videos from images with VACE, Wan2.2, LTX2, +HunyuanVideo-1.5, Cosmos3, and other compatible pipelines. ## Local CLI Usage @@ -68,7 +67,10 @@ python image_to_video.py \ Key arguments: - `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V). -- `--image`: Path to input image (required). +- `--image`: Path to the first-frame or source image. +- `--last-image`: Optional last-frame condition for models such as VACE. +- `--mask-image`: Optional inpainting mask. White pixels are regenerated and black pixels are preserved. +- `--reference-image`: Optional reference image; repeat it to provide multiple references. - `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3. - `--prompt`: Text description of desired motion/animation. - `--height/--width`: Output resolution (auto-calculated from image if not set). Dimensions should be multiples of 16. @@ -94,5 +96,98 @@ Key arguments: > ℹ️ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage. +## Wan2.1 VACE Conditional Tasks + +The shared script selects the VACE conditioning structure from the media inputs. +No explicit mode parameter is required: the script constructs the source video, +mask, or reference images consumed by the VACE pipeline. +Download the same Hugging Face assets used by the original VACE example: + +```bash +wget -O astronaut.jpg https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg +wget -O vace_first_frame.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png +wget -O vace_last_frame.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png +``` + +### Image-to-Video (I2V) + +```bash +python image_to_video.py \ + --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \ + --image astronaut.jpg \ + --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \ + --seed 42 --height 480 --width 832 --num-frames 81 \ + --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \ + --vae-use-tiling --output vace_i2v_output.mp4 +``` + +### Video-to-Last-Frame (V2LF) + +```bash +python image_to_video.py \ + --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \ + --last-image astronaut.jpg \ + --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \ + --seed 42 --height 480 --width 832 --num-frames 81 \ + --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \ + --vae-use-tiling --output vace_v2lf_output.mp4 +``` + +### First-Last-Frame-to-Video (FLF2V) + +```bash +python image_to_video.py \ + --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \ + --image vace_first_frame.png \ + --last-image vace_last_frame.png \ + --prompt "CG animation style, a small blue bird takes off from a branch and lands on another branch" \ + --seed 42 --height 512 --width 512 --num-frames 81 \ + --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \ + --vae-use-tiling --output vace_flf2v_output.mp4 +``` + +### Inpainting + +Create a mask matching the original VACE example: a 160-pixel-wide white +vertical stripe marks the region to regenerate. + +```bash +python - <<'PY' +from PIL import Image + +mask = Image.new("L", (832, 480), 0) +mask.paste(255, (336, 0, 496, 480)) +mask.save("vace_center_mask.png") +PY +``` + +```bash +python image_to_video.py \ + --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \ + --image astronaut.jpg \ + --mask-image vace_center_mask.png \ + --prompt "Shrek, the ogre, walks out of a building in a happy mood" \ + --seed 42 --height 480 --width 832 --num-frames 81 \ + --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \ + --vae-use-tiling --output vace_inpaint_output.mp4 +``` + +### Reference-to-Video (R2V) + +Repeat `--reference-image` to provide more than one reference image. + +```bash +python image_to_video.py \ + --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \ + --reference-image astronaut.jpg \ + --prompt "Camera slowly zooms out from the character walking in a garden" \ + --seed 42 --height 480 --width 832 --num-frames 81 \ + --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \ + --vae-use-tiling --output vace_r2v_output.mp4 +``` + +The VACE T2V command is documented in the shared +[`text_to_video.py`](../text_to_video/text_to_video.md#wan21-vace-t2v) example. + For Wan2.2 LightX2V-converted local Diffusers directories and related LoRA assets, see the [LoRA guide](../../../docs/user_guide/diffusion/lora.md#wan22-lightx2v-offline-assembly). diff --git a/examples/offline_inference/image_to_video/image_to_video.py b/examples/offline_inference/image_to_video/image_to_video.py index 97a01617f..cc8f1ff2a 100644 --- a/examples/offline_inference/image_to_video/image_to_video.py +++ b/examples/offline_inference/image_to_video/image_to_video.py @@ -2,13 +2,14 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """ -Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, or HunyuanVideo-1.5. +Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE. Supports: - Wan2.2-I2V-A14B-Diffusers: MoE model with CLIP image encoder - Wan2.2-TI2V-5B-Diffusers: Unified T2V+I2V model (dense 5B) - LTX2 image-to-video pipeline - HunyuanVideo-1.5 I2V: SigLIP + VAE dual image conditioning +- Wan2.1 VACE: first/last-frame, inpainting, and reference conditioning Usage: # Wan I2V-A14B (MoE) @@ -53,7 +54,11 @@ from vllm_omni.diffusion.data import DiffusionParallelConfig from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args from vllm_omni.entrypoints.omni import Omni from vllm_omni.inputs.data import OmniDiffusionSamplingParams -from vllm_omni.model_extras import get_extra_body_params, get_model_class_name +from vllm_omni.model_extras import ( + build_image_to_video_prompt, + get_extra_body_params, + get_model_class_name, +) from vllm_omni.outputs import OmniRequestOutput from vllm_omni.platforms import current_omni_platform @@ -73,18 +78,28 @@ parse_profiler_config = functools.partial(parse_json_object, flag_name="--profil def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description="Generate a video from an image (Wan2.2, LTX2, HunyuanVideo-1.5).") + parser = argparse.ArgumentParser( + description="Generate a video from one or more images (Wan2.2, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE)." + ) parser.add_argument( "--model", default="Wan-AI/Wan2.2-I2V-A14B-Diffusers", - help="Diffusers I2V model ID or local path (Wan2.2 or HunyuanVideo-1.5).", + help="Diffusers I2V model ID or local path (Wan2.2, HunyuanVideo-1.5, or Wan2.1 VACE).", ) parser.add_argument( "--model-class-name", default=None, help="Override model class name (e.g., LTX2ImageToVideoPipeline).", ) - parser.add_argument("--image", required=True, help="Path to input image.") + parser.add_argument("--image", help="Path to the first-frame or source image.") + parser.add_argument("--last-image", help="Path to a last-frame condition (used by models such as VACE).") + parser.add_argument("--mask-image", help="Path to an inpainting mask (used by models such as VACE).") + parser.add_argument( + "--reference-image", + action="append", + default=None, + help="Path to a reference image. Repeat to provide multiple references.", + ) parser.add_argument("--prompt", default="", help="Text prompt describing the desired motion.") parser.add_argument("--negative-prompt", default="", help="Negative prompt.") parser.add_argument("--seed", type=int, default=42, help="Random seed.") @@ -302,8 +317,15 @@ def main(): if model_class_name is None and is_ltx2: model_class_name = "LTX2ImageToVideoPipeline" - # Load input image - image = PIL.Image.open(args.image).convert("RGB") + image = PIL.Image.open(args.image).convert("RGB") if args.image else None + last_image = PIL.Image.open(args.last_image).convert("RGB") if args.last_image else None + mask_image = PIL.Image.open(args.mask_image).convert("L") if args.mask_image else None + reference_images = ( + [PIL.Image.open(path).convert("RGB") for path in args.reference_image] if args.reference_image else None + ) + dimension_image = image or last_image or (reference_images[0] if reference_images else None) + if dimension_image is None: + raise ValueError("Provide --image, --last-image, or at least one --reference-image.") # Per-model generation defaults, applied only when the matching flag is omitted. # Cosmos3 would otherwise silently inherit the Wan2.2 defaults (wrong size/steps/shift). @@ -333,12 +355,21 @@ def main(): height = args.height width = args.width if height is None or width is None: - calc_height, calc_width = calculate_dimensions(image, max_area=d_max_area, mod_value=d_mod) + calc_height, calc_width = calculate_dimensions(dimension_image, max_area=d_max_area, mod_value=d_mod) height = height or calc_height width = width or calc_width - # Resize image to target dimensions - image = image.resize((width, height), PIL.Image.Resampling.LANCZOS) + media_inputs: dict[str, Any] = {} + if image is not None: + media_inputs["image"] = image.resize((width, height), PIL.Image.Resampling.LANCZOS) + if last_image is not None: + media_inputs["last_image"] = last_image.resize((width, height), PIL.Image.Resampling.LANCZOS) + if mask_image is not None: + media_inputs["mask"] = mask_image.resize((width, height), PIL.Image.Resampling.NEAREST) + if reference_images is not None: + media_inputs["reference_images"] = [ + reference.resize((width, height), PIL.Image.Resampling.LANCZOS) for reference in reference_images + ] # Configure cache based on backend type cache_config = None @@ -409,8 +440,8 @@ def main(): if args.extra_body and "guardrails" in args.extra_body: omni_kwargs["model_config"] = {"guardrails": bool(args.extra_body["guardrails"])} omni = Omni(**omni_kwargs) - resolved_model_class_name = get_model_class_name(omni) - declared_extra_body_params = get_extra_body_params(resolved_model_class_name) + model_class_name = get_model_class_name(omni) or model_class_name + declared_extra_body_params = get_extra_body_params(model_class_name) if profiler_enabled: print("[Profiler] Starting profiling...") @@ -434,6 +465,15 @@ def main(): print(f" Video size: {args.width}x{args.height}") print(f"{'=' * 60}\n") + prompt_dict = build_image_to_video_prompt( + model_class_name=model_class_name, + prompt=args.prompt, + negative_prompt=args.negative_prompt, + media_inputs=media_inputs, + height=height, + width=width, + num_frames=num_frames, + ) sampling_params = OmniDiffusionSamplingParams( height=height, width=width, @@ -461,14 +501,7 @@ def main(): generation_start = time.perf_counter() # omni.generate() returns Generator[OmniRequestOutput, None, None] - frames = omni.generate( - { - "prompt": args.prompt, - "negative_prompt": args.negative_prompt, - "multi_modal_data": {"image": image}, - }, - sampling_params, - ) + frames = omni.generate(prompt_dict, sampling_params) generation_end = time.perf_counter() generation_time = generation_end - generation_start diff --git a/examples/offline_inference/text_to_video/text_to_video.md b/examples/offline_inference/text_to_video/text_to_video.md index 903a514d0..a2634d5ae 100644 --- a/examples/offline_inference/text_to_video/text_to_video.md +++ b/examples/offline_inference/text_to_video/text_to_video.md @@ -6,6 +6,7 @@ A unified script for text-to-video generation. Supports multiple models with mod | Model | Default Resolution | Default Frames | Default Steps | Guidance | VRAM (BF16) | |---|---|---|---|---|---| +| `Wan-AI/Wan2.1-VACE-1.3B-diffusers` | 480x832 | 81 | 30 | 5.0 | ~20 GiB (RTX 5090, VAE tiling) | | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB | | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB | | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required | @@ -31,6 +32,28 @@ python text_to_video.py \ --output t2v_out.mp4 ``` +### Wan2.1 VACE (T2V) + +VACE text-to-video uses this shared entrypoint. Conditional VACE tasks use +the shared [`image_to_video.py`](../image_to_video/README.md#wan21-vace-conditional-tasks) +entrypoint, which constructs the pipeline-native conditioning data from the +provided media inputs. No explicit mode parameter is required. + +```bash +python text_to_video.py \ + --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \ + --prompt "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves." \ + --seed 0 \ + --height 480 \ + --width 832 \ + --num-frames 81 \ + --num-inference-steps 30 \ + --guidance-scale 5.0 \ + --flow-shift 5.0 \ + --vae-use-tiling \ + --output vace_t2v_output.mp4 +``` + LTX2 example: ```bash diff --git a/examples/offline_inference/text_to_video/text_to_video.py b/examples/offline_inference/text_to_video/text_to_video.py index e3a227bcd..1323985e5 100644 --- a/examples/offline_inference/text_to_video/text_to_video.py +++ b/examples/offline_inference/text_to_video/text_to_video.py @@ -19,6 +19,15 @@ from vllm_omni.outputs import OmniRequestOutput from vllm_omni.platforms import current_omni_platform _MODEL_PRESETS = { + "vace": { + "height": 480, + "width": 832, + "num_frames": 81, + "num_inference_steps": 30, + "guidance_scale": 5.0, + "fps": 16, + "output": "vace_t2v_output.mp4", + }, "wan": { "height": 720, "width": 1280, @@ -61,6 +70,8 @@ _MODEL_PRESETS = { def _detect_preset(model: str) -> dict: model_lower = model.lower() + if "vace" in model_lower: + return _MODEL_PRESETS["vace"] if "cosmos" in model_lower: return _MODEL_PRESETS["cosmos"] if "hunyuan" in model_lower: @@ -381,7 +392,7 @@ def main(): omni_kwargs["model_config"] = {"guardrails": bool(args.extra_body["guardrails"])} omni = Omni(**omni_kwargs) - model_class_name = get_model_class_name(omni) + model_class_name = get_model_class_name(omni) or model_class_name declared_extra_body_params = get_extra_body_params(model_class_name) if profiler_enabled: @@ -419,7 +430,6 @@ def main(): sampling_kwargs["guidance_scale_2"] = args.guidance_scale_high sampling_params = OmniDiffusionSamplingParams(**sampling_kwargs) - # Route model-specific knobs through extra_body, filtered against the model's # declared extra_body_params. Models without a declaration only forward explicit # --extra-body JSON (preserving the generic flags' legacy behavior). diff --git a/examples/offline_inference/vace/vace_video_generation.md b/examples/offline_inference/vace/vace_video_generation.md deleted file mode 100644 index 8d8b00e48..000000000 --- a/examples/offline_inference/vace/vace_video_generation.md +++ /dev/null @@ -1,148 +0,0 @@ -# VACE Video Generation - -Generate videos from text prompts, images, or video conditions using vLLM-Omni's VACE diffusion pipeline. - -- `vace_video_generation.py`: command-line script for multi-mode video generation with advanced options. - -## Table of Contents - -- [Overview](#overview) -- [Quick Start](#quick-start) -- [Key Arguments](#key-arguments) -- [More CLI Examples](#more-cli-examples) - -## Overview - -[VACE](https://github.com/ali-vilab/VACE) (Video All-in-one Creation Engine) supports multiple video tasks through a single unified model, including text-to-video, image-to-video, first-last-frame interpolation, inpainting, and reference image-guided generation. - -### Supported Models - -| Model | Architecture | Peak VRAM (GiB) * | Model Weights (GiB) | HuggingFace | -|-------|-------------|-------------------|---------------------|-------------| -| Wan2.1-VACE (1.3B) | Wan2.1 | TBD | ~10 | [Wan-AI/Wan2.1-VACE-1.3B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers) | -| Wan2.1-VACE (14B) | Wan2.1 | TBD | ~38 | [Wan-AI/Wan2.1-VACE-14B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) | - -!!! info -*Peak VRAM: based on basic single-card usage, 480x832 resolution, 81 frames, 30 inference steps, without any acceleration/optimization features. - -Default model: `Wan-AI/Wan2.1-VACE-14B-diffusers` - -## Quick Start - -### Python API - -Text-to-video generation: - -```python -import torch - -from vllm_omni.entrypoints.omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -if __name__ == "__main__": - omni = Omni(model="Wan-AI/Wan2.1-VACE-1.3B-diffusers") - prompt = "A sleek robot stands in a vast warehouse filled with boxes" - outputs = omni.generate( - prompt, - OmniDiffusionSamplingParams( - height=480, - width=832, - num_frames=81, - num_inference_steps=30, - guidance_scale=5.0, - ), - ) - video = outputs[0].images - - from diffusers.utils import export_to_video - export_to_video(list(video[0]), "t2v_output.mp4", fps=16) - omni.close() -``` - -### Local CLI Usage - -```bash -python vace_video_generation.py \ - --mode t2v \ - --prompt "A sleek robot stands in a vast warehouse filled with boxes" \ - --height 480 --width 832 --num-frames 81 \ - --output t2v_output.mp4 -``` - -## Key Arguments - -| Argument | Type | Default | Description | -| -------- | ---- | ------- | ----------- | -| `--mode` | str | `"t2v"` | VACE task mode: `t2v`, `i2v`, `v2lf`, `flf2v`, `inpaint`, `r2v` | -| `--model` | str | `"Wan-AI/Wan2.1-VACE-14B-diffusers"` | Model ID or local path | -| `--prompt` | str | `"A cat walking in a garden"` | Text description of desired video | -| `--negative-prompt` | str | `""` | Negative prompt for classifier-free guidance | -| `--image` | str | `None` | Input image path (for I2V, R2V, FLF2V, inpaint modes) | -| `--last-image` | str | `None` | Last frame image path (for FLF2V mode) | -| `--height` | int | `480` | Output video height in pixels (should be a multiple of 16) | -| `--width` | int | `832` | Output video width in pixels (should be a multiple of 16) | -| `--num-frames` | int | `81` | Number of video frames to generate | -| `--num-inference-steps` | int | `30` | Number of denoising steps (more steps = higher quality, slower) | -| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale | -| `--flow-shift` | float | `5.0` | Scheduler flow shift parameter | -| `--seed` | int | `42` | Random seed for deterministic sampling | -| `--fps` | int | `16` | Frames per second for the saved MP4 | -| `--output` | str | `"vace_output.mp4"` | Path to save the generated video | -| `--vae-use-tiling` | flag | on | Enable VAE tiling for memory optimization | -| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel | -| `--ulysses-degree` | int | `1` | Ulysses sequence parallel degree for multi-GPU inference | -| `--ring-degree` | int | `1` | Ring sequence parallel degree for hybrid Ulysses + Ring inference | -| `--tensor-parallel-size` | int | — | Tensor parallel size | -| `--enforce-eager` | flag | off | Disable torch.compile | - -> If you encounter OOM errors, try `--vae-use-tiling` or multi-GPU parallelism options (`--ulysses-degree`, `--cfg-parallel-size`). - -## More CLI Examples - -### Image-to-Video (I2V) - -First frame is kept, remaining frames are generated: - -```bash -python vace_video_generation.py \ - --mode i2v \ - --image astronaut.jpg \ - --prompt "An astronaut emerging from a cracked egg on the moon" \ - --height 480 --width 832 --num-frames 81 \ - --output i2v_output.mp4 -``` - -### First-Last-Frame Interpolation (FLF2V) - -```bash -python vace_video_generation.py \ - --mode flf2v \ - --image first_frame.jpg --last-image last_frame.jpg \ - --prompt "A bird takes off from a branch and lands on another" \ - --height 512 --width 512 --num-frames 81 \ - --output flf2v_output.mp4 -``` - -### Inpainting - -Center vertical stripe is masked and regenerated: - -```bash -python vace_video_generation.py \ - --mode inpaint \ - --image scene.jpg \ - --prompt "Shrek walks out of a building" \ - --height 480 --width 832 --num-frames 81 \ - --output inpaint_output.mp4 -``` - -### Reference Image-guided (R2V) - -```bash -python vace_video_generation.py \ - --mode r2v \ - --image reference.jpg \ - --prompt "Camera slowly zooms out from the character" \ - --height 480 --width 832 --num-frames 81 \ - --output r2v_output.mp4 -``` diff --git a/examples/offline_inference/vace/vace_video_generation.py b/examples/offline_inference/vace/vace_video_generation.py deleted file mode 100644 index 6ca0d74c5..000000000 --- a/examples/offline_inference/vace/vace_video_generation.py +++ /dev/null @@ -1,209 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -"""VACE video generation example. - -VACE (Video All-in-one Creation Engine) supports multiple video tasks: - - T2V: Text-to-Video - - I2V: Image-to-Video (first frame conditioning) - - V2LF: Video-to-Last-Frame - - FLF2V: First-Last-Frame interpolation - - Inpainting: Masked region generation - - R2V: Reference image-guided generation - -Usage examples: - # T2V (text-to-video) - python vace_video_generation.py --mode t2v --prompt "A robot in a warehouse" - - # I2V (image-to-video, first frame kept) - python vace_video_generation.py --mode i2v --image input.jpg --prompt "..." - - # FLF2V (first-last frame interpolation) - python vace_video_generation.py --mode flf2v --image first.jpg --last-image last.jpg - - # R2V (reference image guided) - python vace_video_generation.py --mode r2v --image ref.jpg --prompt "..." -""" - -import argparse -import time -from pathlib import Path - -import numpy as np -import PIL.Image -import torch - -from vllm_omni.diffusion.data import DiffusionParallelConfig -from vllm_omni.entrypoints.omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams -from vllm_omni.platforms import current_omni_platform - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description="VACE video generation.") - parser.add_argument( - "--model", - default="Wan-AI/Wan2.1-VACE-14B-diffusers", - help="VACE model ID or local path.", - ) - parser.add_argument( - "--mode", - default="t2v", - choices=["t2v", "i2v", "v2lf", "flf2v", "inpaint", "r2v"], - help="Generation mode.", - ) - parser.add_argument("--prompt", default="A cat walking in a garden", help="Text prompt.") - parser.add_argument("--negative-prompt", default="", help="Negative prompt.") - parser.add_argument("--image", type=str, default=None, help="Input image path (for I2V, R2V, FLF2V, inpaint).") - parser.add_argument("--last-image", type=str, default=None, help="Last frame image path (for FLF2V).") - parser.add_argument("--video-dir", type=str, default=None, help="Directory of video frames (for inpaint).") - parser.add_argument("--seed", type=int, default=42, help="Random seed.") - parser.add_argument("--guidance-scale", type=float, default=5.0, help="CFG guidance scale.") - parser.add_argument("--height", type=int, default=480, help="Video height.") - parser.add_argument("--width", type=int, default=832, help="Video width.") - parser.add_argument("--num-frames", type=int, default=81, help="Number of frames.") - parser.add_argument("--num-inference-steps", type=int, default=30, help="Sampling steps.") - parser.add_argument("--flow-shift", type=float, default=5.0, help="Scheduler flow_shift.") - parser.add_argument("--output", type=str, default="vace_output.mp4", help="Output video path.") - parser.add_argument("--fps", type=int, default=16, help="Output video FPS.") - parser.add_argument("--vae-use-tiling", action="store_true", default=True, help="Enable VAE tiling.") - parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.") - parser.add_argument("--ulysses-degree", type=int, default=1, help="Ulysses SP degree.") - parser.add_argument("--ring-degree", type=int, default=1, help="Ring attention degree.") - parser.add_argument("--cfg-parallel-size", type=int, default=1, choices=[1, 2], help="CFG parallel size.") - return parser.parse_args() - - -def build_prompts(args): - """Build prompt dict with multi_modal_data based on mode.""" - h, w, nf = args.height, args.width, args.num_frames - - gray = PIL.Image.new("RGB", (w, h), (128, 128, 128)) - mask_black = PIL.Image.new("L", (w, h), 0) - mask_white = PIL.Image.new("L", (w, h), 255) - - prompt_data = { - "prompt": args.prompt, - "negative_prompt": args.negative_prompt, - } - - if args.mode == "t2v": - return prompt_data - - if args.mode == "r2v": - assert args.image, "--image required for R2V mode" - ref_img = PIL.Image.open(args.image).convert("RGB").resize((w, h)) - prompt_data["multi_modal_data"] = {"reference_images": [ref_img]} - return prompt_data - - if args.mode == "i2v": - assert args.image, "--image required for I2V mode" - img = PIL.Image.open(args.image).convert("RGB").resize((w, h)) - prompt_data["multi_modal_data"] = { - "video": [img] + [gray] * (nf - 1), - "mask": [mask_black] + [mask_white] * (nf - 1), - } - return prompt_data - - if args.mode == "v2lf": - assert args.image, "--image required for V2LF mode" - img = PIL.Image.open(args.image).convert("RGB").resize((w, h)) - prompt_data["multi_modal_data"] = { - "video": [gray] * (nf - 1) + [img], - "mask": [mask_white] * (nf - 1) + [mask_black], - } - return prompt_data - - if args.mode == "flf2v": - assert args.image and args.last_image, "--image and --last-image required for FLF2V" - first = PIL.Image.open(args.image).convert("RGB").resize((w, h)) - last = PIL.Image.open(args.last_image).convert("RGB").resize((w, h)) - prompt_data["multi_modal_data"] = { - "video": [first] + [gray] * (nf - 2) + [last], - "mask": [mask_black] + [mask_white] * (nf - 2) + [mask_black], - } - return prompt_data - - if args.mode == "inpaint": - assert args.image, "--image required for inpaint mode" - img = PIL.Image.open(args.image).convert("RGB").resize((w, h)) - d = 80 - frames, masks = [], [] - for _ in range(nf): - base = np.array(img).copy() - mask = PIL.Image.new("L", (w, h), 0) - stripe = PIL.Image.new("L", (2 * d, h), 255) - mask.paste(stripe, (w // 2 - d, 0)) - base[np.array(mask) > 128] = 128 - frames.append(PIL.Image.fromarray(base)) - masks.append(mask) - prompt_data["multi_modal_data"] = {"video": frames, "mask": masks} - return prompt_data - - raise ValueError(f"Unknown mode: {args.mode}") - - -def main(): - args = parse_args() - generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed) - - parallel_config = DiffusionParallelConfig( - ulysses_degree=args.ulysses_degree, - ring_degree=args.ring_degree, - cfg_parallel_size=args.cfg_parallel_size, - ) - - omni = Omni( - model=args.model, - vae_use_tiling=args.vae_use_tiling, - flow_shift=args.flow_shift, - enforce_eager=args.enforce_eager, - parallel_config=parallel_config, - ) - - prompt_data = build_prompts(args) - - print(f"\n{'=' * 60}") - print(f"VACE {args.mode.upper()} Generation") - print(f" Model: {args.model}") - print(f" Size: {args.width}x{args.height}, {args.num_frames} frames, {args.num_inference_steps} steps") - print(f"{'=' * 60}\n") - - start = time.perf_counter() - outputs = omni.generate( - prompt_data, - OmniDiffusionSamplingParams( - height=args.height, - width=args.width, - num_frames=args.num_frames, - num_inference_steps=args.num_inference_steps, - guidance_scale=args.guidance_scale, - generator=generator, - ), - ) - elapsed = time.perf_counter() - start - - video = outputs[0].images - if isinstance(video, list): - video = video[0] - if isinstance(video, torch.Tensor): - video = video.cpu().numpy() - if video.ndim == 5: - video = video[0] - print(f"Output shape: {video.shape}, Time: {elapsed:.1f}s") - - output_path = Path(args.output) - output_path.parent.mkdir(parents=True, exist_ok=True) - - from diffusers.utils import export_to_video - - if np.issubdtype(video.dtype, np.integer): - video = video.astype(np.float32) / 255.0 - export_to_video(list(video), str(output_path), fps=args.fps) - print(f"Saved to {output_path}") - - omni.close() - - -if __name__ == "__main__": - main() diff --git a/recipes/README.md b/recipes/README.md index 67d481860..63dd74174 100644 --- a/recipes/README.md +++ b/recipes/README.md @@ -63,6 +63,7 @@ recipes/ | [`Tencent/HunyuanImage-3.0-Instruct.md`](./Tencent/HunyuanImage-3.0-Instruct.md) | DiT-only text-to-image serving and benchmark, including ModelOpt mixed FP8/NVFP4 | 4x H100/H800 80GB / 2x B200 | | [`Wan-AI/Wan2.2-I2V.md`](./Wan-AI/Wan2.2-I2V.md) | Image-to-video serving (Wan2.2 14B) | 8x Ascend NPU (A2/A3) | | [`Wan-AI/Wan2.2-S2V.md`](./Wan-AI/Wan2.2-S2V.md) | Speech-to-video serving (Wan2.2 14B) | 2x A100/H100 80GB | +| [`Wan-AI/Wan2.1-VACE.md`](./Wan-AI/Wan2.1-VACE.md) | Unified T2V, I2V, V2LF, FLF2V, inpaint, and R2V | 1x RTX 5090 (1.3B) / 1x L40S 48GB with layerwise offload (14B) | | [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB | | [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB | | [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB | diff --git a/recipes/Wan-AI/Wan2.1-VACE.md b/recipes/Wan-AI/Wan2.1-VACE.md new file mode 100644 index 000000000..c187e5455 --- /dev/null +++ b/recipes/Wan-AI/Wan2.1-VACE.md @@ -0,0 +1,147 @@ +# Wan2.1 VACE Unified Video Generation + +## Summary + +- Vendor: Wan-AI +- Models: `Wan2.1-VACE-1.3B-diffusers`, `Wan2.1-VACE-14B-diffusers` +- Tasks: T2V, I2V, V2LF, FLF2V, inpainting, and R2V +- Mode: offline inference through the shared task examples + +The shared examples construct VACE's pipeline-native conditioning data from the +provided media inputs. They do not require a mode parameter or a model-specific +example script. + +## References + +- [VACE project](https://github.com/ali-vilab/VACE) +- [Hugging Face 14B checkpoint](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) +- [ModelScope 14B checkpoint](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B-diffusers) +- [Shared text-to-video example](../../examples/offline_inference/text_to_video/text_to_video.md) +- [Shared image-to-video example](../../examples/offline_inference/image_to_video/README.md) +- [Migration RFC #4539](https://github.com/vllm-project/vllm-omni/issues/4539) + +## Hardware Support + +The 1.3B checkpoint was validated on an RTX 5090; the 81-frame T2V command below +used approximately 20 GiB of peak reserved GPU memory with VAE tiling. The 14B +checkpoint requires approximately 70 GiB of disk space and was validated on a +single 48 GiB L40S with VAE tiling and layerwise offload. Start with the smoke +configuration before attempting an 81-frame run. + +## Setup + +The public examples use the Hugging Face model ID by default: + +```bash +MODEL="${MODEL:-Wan-AI/Wan2.1-VACE-14B-diffusers}" +ASSET_DIR="${ASSET_DIR:-$HOME/datasets/vace}" +OUT="${OUT:-$PWD/vace-results/14b}" +mkdir -p "$ASSET_DIR" "$OUT" +``` + +For environments where Hugging Face downloads are unavailable, download the +same checkpoint from ModelScope and point `MODEL` at the local directory: + +```bash +MODEL="$HOME/models/Wan2.1-VACE-14B-diffusers" +modelscope download --model Wan-AI/Wan2.1-VACE-14B-diffusers --local_dir "$MODEL" +``` + +Download the public conditioning images and create a mask matching the output +size. White pixels are regenerated and black pixels are preserved. + +```bash +wget -O "$ASSET_DIR/astronaut.jpg" \ + https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg +wget -O "$ASSET_DIR/vace_first_frame.png" \ + https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png +wget -O "$ASSET_DIR/vace_last_frame.png" \ + https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png + +python - "$ASSET_DIR/vace_center_mask.png" <<'PY' +import sys +from PIL import Image + +mask = Image.new("L", (832, 480), 0) +mask.paste(255, (336, 0, 496, 480)) +mask.save(sys.argv[1]) +PY + +ASTRONAUT="$ASSET_DIR/astronaut.jpg" +FIRST="$ASSET_DIR/vace_first_frame.png" +LAST="$ASSET_DIR/vace_last_frame.png" +MASK="$ASSET_DIR/vace_center_mask.png" +``` + +## Commands + +The commands below reproduce the original VACE examples with `81 frames / 30 +steps`. Most tasks use `480x832`; FLF2V uses `512x512`. For a smoke test, +replace the frame and step counts with `5 frames / 2 steps`. + +```bash +# T2V +python examples/offline_inference/text_to_video/text_to_video.py \ + --model "$MODEL" \ + --prompt "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves." \ + --seed 0 \ + --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \ + --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \ + --output "$OUT/t2v.mp4" + +# I2V +python examples/offline_inference/image_to_video/image_to_video.py \ + --model "$MODEL" --image "$ASTRONAUT" \ + --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \ + --seed 42 \ + --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \ + --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \ + --output "$OUT/i2v.mp4" + +# V2LF +python examples/offline_inference/image_to_video/image_to_video.py \ + --model "$MODEL" --last-image "$ASTRONAUT" \ + --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \ + --seed 42 \ + --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \ + --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \ + --output "$OUT/v2lf.mp4" + +# FLF2V +python examples/offline_inference/image_to_video/image_to_video.py \ + --model "$MODEL" --image "$FIRST" --last-image "$LAST" \ + --prompt "CG animation style, a small blue bird takes off from a branch and lands on another branch" \ + --seed 42 --height 512 --width 512 --num-frames 81 --num-inference-steps 30 \ + --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \ + --output "$OUT/flf2v.mp4" + +# Inpaint +python examples/offline_inference/image_to_video/image_to_video.py \ + --model "$MODEL" --image "$ASTRONAUT" --mask-image "$MASK" \ + --prompt "Shrek, the ogre, walks out of a building in a happy mood" \ + --seed 42 \ + --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \ + --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \ + --output "$OUT/inpaint.mp4" + +# R2V; repeat --reference-image to add references +python examples/offline_inference/image_to_video/image_to_video.py \ + --model "$MODEL" --reference-image "$ASTRONAUT" \ + --prompt "Camera slowly zooms out from the character walking in a garden" \ + --seed 42 \ + --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \ + --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \ + --output "$OUT/r2v.mp4" +``` + +## Verification + +```bash +for video in "$OUT"/*.mp4; do + ffprobe -v error -select_streams v:0 \ + -show_entries stream=codec_name,width,height,nb_frames,r_frame_rate,duration \ + -of default=noprint_wrappers=1 "$video" +done +``` + +For the 1.3B checkpoint, `--enable-layerwise-offload` is generally unnecessary. diff --git a/tests/examples/offline_inference/test_vace_video_generation.py b/tests/examples/offline_inference/test_vace_video_generation.py index 11746e58a..e4f6ca93d 100644 --- a/tests/examples/offline_inference/test_vace_video_generation.py +++ b/tests/examples/offline_inference/test_vace_video_generation.py @@ -1,11 +1,9 @@ -""" -Offline inference tests: vace video generation. -See examples/offline_inference/vace/vace_video_generation.md -""" +"""VACE coverage through the shared text-to-video and image-to-video examples.""" from pathlib import Path import pytest +from PIL import Image from tests.examples.helpers import EXAMPLES, ExampleRunner, ReadmeSnippet from tests.helpers.assertions import assert_video_valid @@ -14,22 +12,83 @@ from tests.helpers.mark import hardware_marks pytestmark = [ pytest.mark.usefixtures("clean_gpu_memory_between_tests"), pytest.mark.full_model, + pytest.mark.diffusion, pytest.mark.example, *hardware_marks(res={"cuda": "H100"}), ] -VACE_SCRIPT = EXAMPLES / "offline_inference" / "vace" / "vace_video_generation.py" -README_PATH = VACE_SCRIPT.with_name("vace_video_generation.md") +TEXT_TO_VIDEO_README = EXAMPLES / "offline_inference" / "text_to_video" / "text_to_video.md" +IMAGE_TO_VIDEO_README = EXAMPLES / "offline_inference" / "image_to_video" / "README.md" EXAMPLE_OUTPUT_SUBFOLDER = "example_offline_vace" -README_SNIPPETS = ReadmeSnippet.extract_readme_snippets(README_PATH) +TEXT_TO_VIDEO_SNIPPET = next( + snippet + for snippet in ReadmeSnippet.extract_readme_snippets(TEXT_TO_VIDEO_README) + if "vace_t2v_output.mp4" in snippet.code +) +CONDITIONAL_OUTPUTS = ( + "vace_i2v_output.mp4", + "vace_v2lf_output.mp4", + "vace_flf2v_output.mp4", + "vace_inpaint_output.mp4", + "vace_r2v_output.mp4", +) +IMAGE_TO_VIDEO_SNIPPETS = [ + snippet + for snippet in ReadmeSnippet.extract_readme_snippets(IMAGE_TO_VIDEO_README) + if snippet.h2_title == "Wan2.1 VACE Conditional Tasks" + and any(output in snippet.code for output in CONDITIONAL_OUTPUTS) +] -@pytest.mark.parametrize("snippet", README_SNIPPETS, ids=lambda snippet: snippet.test_id) -def test_vace_video_generation(snippet: ReadmeSnippet, example_runner: ExampleRunner): - should_skip, reason = snippet.skip - if should_skip: - pytest.skip(reason) +def _smoke_snippet(snippet: ReadmeSnippet, asset_dir: Path) -> ReadmeSnippet: + # Six fresh model launches share a 90-minute X2V job. Keep the documented + # quality settings intact while reducing only the CI executions. + replacements = { + "astronaut.jpg": str(asset_dir / "astronaut.jpg"), + "vace_first_frame.png": str(asset_dir / "vace_first_frame.png"), + "vace_last_frame.png": str(asset_dir / "vace_last_frame.png"), + "vace_center_mask.png": str(asset_dir / "vace_center_mask.png"), + "--num-frames 81": "--num-frames 5", + "--num-inference-steps 30": "--num-inference-steps 2", + } + code = snippet.code + for source, target in replacements.items(): + code = code.replace(source, target) + return snippet._replace(code=code) + + +@pytest.fixture +def vace_asset_dir(tmp_path: Path) -> Path: + asset_dir = tmp_path / "vace_assets" + asset_dir.mkdir(parents=True, exist_ok=True) + Image.new("RGB", (832, 480), (180, 180, 180)).save(asset_dir / "astronaut.jpg") + Image.new("RGB", (512, 512), (80, 140, 220)).save(asset_dir / "vace_first_frame.png") + Image.new("RGB", (512, 512), (220, 140, 80)).save(asset_dir / "vace_last_frame.png") + mask = Image.new("L", (832, 480), 0) + mask.paste(255, (336, 0, 496, 480)) + mask.save(asset_dir / "vace_center_mask.png") + return asset_dir + + +def test_vace_text_to_video_shared_example(example_runner: ExampleRunner) -> None: + snippet = TEXT_TO_VIDEO_SNIPPET._replace( + code=TEXT_TO_VIDEO_SNIPPET.code.replace("--num-frames 81", "--num-frames 5").replace( + "--num-inference-steps 30", "--num-inference-steps 2" + ) + ) + result = example_runner.run(snippet, output_subfolder=Path(EXAMPLE_OUTPUT_SUBFOLDER)) + for asset in result.assets: + assert_video_valid(asset) + + +@pytest.mark.parametrize("snippet", IMAGE_TO_VIDEO_SNIPPETS, ids=lambda snippet: snippet.test_id) +def test_vace_conditional_shared_examples( + snippet: ReadmeSnippet, + example_runner: ExampleRunner, + vace_asset_dir: Path, +) -> None: + snippet = _smoke_snippet(snippet, vace_asset_dir) result = example_runner.run(snippet, output_subfolder=Path(EXAMPLE_OUTPUT_SUBFOLDER)) for asset in result.assets: assert_video_valid(asset) diff --git a/tests/model_extras/test_model_extras.py b/tests/model_extras/test_model_extras.py index ca34add96..7eaf3edf6 100644 --- a/tests/model_extras/test_model_extras.py +++ b/tests/model_extras/test_model_extras.py @@ -10,6 +10,7 @@ from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args from vllm_omni.inputs.data import OmniDiffusionSamplingParams from vllm_omni.model_extras import ( build_image_to_image_prompt, + build_image_to_video_prompt, build_text_to_image_prompt, get_extra_body_params, get_extra_output_params, @@ -176,6 +177,14 @@ def test_helios_extra_registry_declares_request_and_response_params() -> None: assert should_init_extra_args_for_non_diffusion_stages(cls) is False +@pytest.mark.core_model +@pytest.mark.cpu +def test_vace_extra_registry_has_no_pipeline_params() -> None: + assert get_extra_body_params("WanVACEPipeline") == frozenset() + assert get_extra_output_params("WanVACEPipeline") == frozenset() + assert should_init_extra_args_for_non_diffusion_stages("WanVACEPipeline") is False + + @pytest.mark.core_model @pytest.mark.cpu def test_unknown_pipeline_has_empty_extra_registry() -> None: @@ -253,6 +262,58 @@ def test_unknown_pipeline_uses_default_image_to_image_prompt() -> None: } +def _build_vace_prompt(media_inputs: dict[str, object], *, num_frames: int = 5) -> dict: + return build_image_to_video_prompt( + "WanVACEPipeline", + prompt="a bird flying", + negative_prompt=None, + media_inputs=media_inputs, + height=16, + width=320, + num_frames=num_frames, + ) + + +@pytest.mark.core_model +@pytest.mark.cpu +@pytest.mark.parametrize( + "media_inputs", + [ + {"image": Image.new("RGB", (320, 16), "red")}, + {"last_image": Image.new("RGB", (320, 16), "blue")}, + { + "image": Image.new("RGB", (320, 16), "red"), + "last_image": Image.new("RGB", (320, 16), "blue"), + }, + {"image": Image.new("RGB", (320, 16), "red"), "mask": Image.new("L", (320, 16), 0)}, + {"reference_images": [Image.new("RGB", (64, 64), "red")]}, + ], + ids=["i2v", "v2lf", "flf2v", "inpaint", "r2v"], +) +def test_vace_image_to_video_prompt_builder(media_inputs: dict[str, object]) -> None: + result = _build_vace_prompt(media_inputs) + mmd = result["multi_modal_data"] + if "reference_images" in mmd: + assert mmd["reference_images"] is media_inputs["reference_images"] + else: + assert len(mmd["video"]) == len(mmd["mask"]) == 5 + + +@pytest.mark.core_model +@pytest.mark.cpu +@pytest.mark.parametrize( + ("media_inputs", "message"), + [ + ({}, "requires a conditioning media input"), + ({"mask": Image.new("L", (320, 16))}, "mask input requires an image"), + ({"control_image": Image.new("RGB", (320, 16))}, "Unsupported VACE media input"), + ], +) +def test_vace_rejects_invalid_media_combinations(media_inputs: dict[str, object], message: str) -> None: + with pytest.raises(ValueError, match=message): + _build_vace_prompt(media_inputs) + + @pytest.mark.core_model @pytest.mark.cpu def test_declared_extra_args_apply_to_existing_sampling_params() -> None: diff --git a/vllm_omni/model_extras/__init__.py b/vllm_omni/model_extras/__init__.py index e2f98cd41..5311f495b 100644 --- a/vllm_omni/model_extras/__init__.py +++ b/vllm_omni/model_extras/__init__.py @@ -3,6 +3,7 @@ from vllm_omni.model_extras.registry import ( build_image_to_image_prompt, + build_image_to_video_prompt, build_text_to_image_prompt, get_extra_body_params, get_extra_output_params, @@ -12,6 +13,7 @@ from vllm_omni.model_extras.registry import ( __all__ = [ "build_image_to_image_prompt", + "build_image_to_video_prompt", "build_text_to_image_prompt", "get_extra_body_params", "get_extra_output_params", diff --git a/vllm_omni/model_extras/registry.py b/vllm_omni/model_extras/registry.py index 0b06ce10d..69608764a 100644 --- a/vllm_omni/model_extras/registry.py +++ b/vllm_omni/model_extras/registry.py @@ -3,7 +3,7 @@ from __future__ import annotations -from collections.abc import Callable +from collections.abc import Callable, Mapping from typing import Any from PIL import Image @@ -38,9 +38,14 @@ from vllm_omni.model_extras.sensenova_u1 import ( SENSENOVA_U1_EXTRA_BODY_PARAMS, SENSENOVA_U1_EXTRA_OUTPUT_PARAMS, ) +from vllm_omni.model_extras.vace import ( + VACE_EXTRA_BODY_PARAMS, + VACE_EXTRA_OUTPUT_PARAMS, +) +from vllm_omni.model_extras.vace import ( + build_image_to_video_prompt as build_vace_image_to_video_prompt, +) -# TODO: extend with TextToVideoPromptBuilder, ImageToVideoPromptBuilder -# once a video model needs model-specific prompt construction (follow-up PR). TextToImagePromptBuilder = Callable[ [str, str | None, int | None, int | None], dict[str, Any], @@ -49,6 +54,17 @@ ImageToImagePromptBuilder = Callable[ [str, str | None, "Image.Image | list[Image.Image]", int | None, int | None], dict[str, Any], ] +ImageToVideoPromptBuilder = Callable[ + [ + str, + str | None, + "Mapping[str, Any]", + int | None, + int | None, + int | None, + ], + dict[str, Any], +] def default_text_to_image_prompt( @@ -79,6 +95,20 @@ def default_image_to_image_prompt( return result +def default_image_to_video_prompt( + prompt: str, + negative_prompt: str | None, + media_inputs: Mapping[str, Any], + height: int | None = None, + width: int | None = None, + num_frames: int | None = None, +) -> dict[str, Any]: + del height, width, num_frames + if set(media_inputs) != {"image"} or not isinstance(media_inputs["image"], Image.Image): + raise ValueError("This model only supports a single --image input in the shared image-to-video example.") + return default_image_to_image_prompt(prompt, negative_prompt, media_inputs["image"]) + + _EXTRA_SPECS: dict[str, dict[str, Any]] = { "BagelPipeline": { "extra_body_params": BAGEL_EXTRA_BODY_PARAMS, @@ -108,6 +138,11 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = { "extra_body_params": HELIOS_EXTRA_BODY_PARAMS, "extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS, }, + "WanVACEPipeline": { + "extra_body_params": VACE_EXTRA_BODY_PARAMS, + "extra_output_params": VACE_EXTRA_OUTPUT_PARAMS, + "image_to_video_prompt_builder": build_vace_image_to_video_prompt, + }, } @@ -177,3 +212,21 @@ def build_image_to_image_prompt( else default_image_to_image_prompt ) return builder(prompt, negative_prompt, input_image, height, width) + + +def build_image_to_video_prompt( + model_class_name: str | None, + prompt: str, + negative_prompt: str | None, + media_inputs: Mapping[str, Any], + height: int | None = None, + width: int | None = None, + num_frames: int | None = None, +) -> dict[str, Any]: + spec = _get_spec(model_class_name) + builder: ImageToVideoPromptBuilder = ( + spec.get("image_to_video_prompt_builder", default_image_to_video_prompt) + if spec is not None + else default_image_to_video_prompt + ) + return builder(prompt, negative_prompt, media_inputs, height, width, num_frames) diff --git a/vllm_omni/model_extras/vace.py b/vllm_omni/model_extras/vace.py new file mode 100644 index 000000000..ed626afbe --- /dev/null +++ b/vllm_omni/model_extras/vace.py @@ -0,0 +1,120 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +import numpy as np +from PIL import Image + +# VACE conditioning is expressed through prompt media, and the pipeline does not +# currently read any model-specific values from sampling_params.extra_args. +VACE_EXTRA_BODY_PARAMS: frozenset[str] = frozenset() +VACE_EXTRA_OUTPUT_PARAMS: frozenset[str] = frozenset() + +_MEDIA_KEYS = frozenset({"image", "last_image", "mask", "reference_images"}) + + +def _base_prompt(prompt: str, negative_prompt: str | None) -> dict[str, Any]: + result: dict[str, Any] = {"prompt": prompt} + if negative_prompt is not None: + result["negative_prompt"] = negative_prompt + return result + + +def _require_dimensions( + height: int | None, + width: int | None, + num_frames: int | None, +) -> tuple[int, int, int]: + if height is None or width is None or num_frames is None: + raise ValueError("VACE conditional inputs require height, width, and num_frames.") + if height <= 0 or width <= 0 or num_frames <= 0: + raise ValueError("VACE height, width, and num_frames must be positive.") + return height, width, num_frames + + +def _validate_media_inputs(media_inputs: Mapping[str, Any]) -> None: + unknown = set(media_inputs) - _MEDIA_KEYS + if unknown: + raise ValueError(f"Unsupported VACE media input(s): {', '.join(sorted(unknown))}.") + + has_references = bool(media_inputs.get("reference_images")) + has_frame_conditioning = any(media_inputs.get(key) is not None for key in ("image", "last_image", "mask")) + if has_references and has_frame_conditioning: + raise ValueError("VACE reference images cannot be combined with image, last_image, or mask inputs.") + if media_inputs.get("mask") is not None and media_inputs.get("image") is None: + raise ValueError("VACE mask input requires an image input.") + if media_inputs.get("mask") is not None and media_inputs.get("last_image") is not None: + raise ValueError("VACE mask and last_image inputs cannot be combined.") + if not has_references and not has_frame_conditioning: + raise ValueError("VACE image-to-video generation requires a conditioning media input.") + + +def build_image_to_video_prompt( + prompt: str, + negative_prompt: str | None, + media_inputs: Mapping[str, Any], + height: int | None = None, + width: int | None = None, + num_frames: int | None = None, +) -> dict[str, Any]: + """Build VACE's native video, mask, or reference-image conditioning.""" + _validate_media_inputs(media_inputs) + result = _base_prompt(prompt, negative_prompt) + + reference_images = media_inputs.get("reference_images") + if reference_images: + if not isinstance(reference_images, list) or not all( + isinstance(image, Image.Image) for image in reference_images + ): + raise ValueError("VACE reference_images must be a non-empty list of PIL images.") + result["multi_modal_data"] = {"reference_images": reference_images} + return result + + height, width, num_frames = _require_dimensions(height, width, num_frames) + first_image = media_inputs.get("image") + last_image = media_inputs.get("last_image") + mask_image = media_inputs.get("mask") + for name, image in (("image", first_image), ("last_image", last_image), ("mask", mask_image)): + if image is not None and not isinstance(image, Image.Image): + raise ValueError(f"VACE {name} must be a PIL image.") + + gray = Image.new("RGB", (width, height), (128, 128, 128)) + mask_black = Image.new("L", (width, height), 0) + mask_white = Image.new("L", (width, height), 255) + + if mask_image is not None: + mask = mask_image.convert("L") + source = np.asarray(first_image).copy() + source[np.asarray(mask) > 128] = 128 + frame = Image.fromarray(source) + result["multi_modal_data"] = { + "video": [frame.copy() for _ in range(num_frames)], + "mask": [mask.copy() for _ in range(num_frames)], + } + return result + + if first_image is not None and last_image is not None: + if num_frames < 2: + raise ValueError("VACE first/last-frame conditioning requires at least two frames.") + result["multi_modal_data"] = { + "video": [first_image] + [gray] * (num_frames - 2) + [last_image], + "mask": [mask_black] + [mask_white] * (num_frames - 2) + [mask_black], + } + return result + + if first_image is not None: + result["multi_modal_data"] = { + "video": [first_image] + [gray] * (num_frames - 1), + "mask": [mask_black] + [mask_white] * (num_frames - 1), + } + return result + + result["multi_modal_data"] = { + "video": [gray] * (num_frames - 1) + [last_image], + "mask": [mask_white] * (num_frames - 1) + [mask_black], + } + return result