[Refactor] Migrate VACE example to standard task examples + model_extras (#4648)

Signed-off-by: ForestWisdom <zszhang01@gmail.com>

examples/offline_inference/image_to_video/README.md

@@ -1,8 +1,7 @@
 # Image-To-Video
 
-This example demonstrates how to generate videos from images (image-to-video)
-using Wan2.2, LTX2, HunyuanVideo-1.5, and Cosmos3 with vLLM-Omni's offline
-inference API.
+This shared example generates videos from images with VACE, Wan2.2, LTX2,
+HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
 
 ## Local CLI Usage
 
@@ -68,7 +67,10 @@ python image_to_video.py \
 Key arguments:
 
 - `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V).
-- `--image`: Path to input image (required).
+- `--image`: Path to the first-frame or source image.
+- `--last-image`: Optional last-frame condition for models such as VACE.
+- `--mask-image`: Optional inpainting mask. White pixels are regenerated and black pixels are preserved.
+- `--reference-image`: Optional reference image; repeat it to provide multiple references.
 - `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3.
 - `--prompt`: Text description of desired motion/animation.
 - `--height/--width`: Output resolution (auto-calculated from image if not set). Dimensions should be multiples of 16.
@@ -94,5 +96,98 @@ Key arguments:
 
 > ℹ️ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage.
 
+## Wan2.1 VACE Conditional Tasks
+
+The shared script selects the VACE conditioning structure from the media inputs.
+No explicit mode parameter is required: the script constructs the source video,
+mask, or reference images consumed by the VACE pipeline.
+Download the same Hugging Face assets used by the original VACE example:
+
+```bash
+wget -O astronaut.jpg https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg
+wget -O vace_first_frame.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png
+wget -O vace_last_frame.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png
+```
+
+### Image-to-Video (I2V)
+
+```bash
+python image_to_video.py \
+  --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
+  --image astronaut.jpg \
+  --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
+  --seed 42 --height 480 --width 832 --num-frames 81 \
+  --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
+  --vae-use-tiling --output vace_i2v_output.mp4
+```
+
+### Video-to-Last-Frame (V2LF)
+
+```bash
+python image_to_video.py \
+  --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
+  --last-image astronaut.jpg \
+  --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
+  --seed 42 --height 480 --width 832 --num-frames 81 \
+  --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
+  --vae-use-tiling --output vace_v2lf_output.mp4
+```
+
+### First-Last-Frame-to-Video (FLF2V)
+
+```bash
+python image_to_video.py \
+  --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
+  --image vace_first_frame.png \
+  --last-image vace_last_frame.png \
+  --prompt "CG animation style, a small blue bird takes off from a branch and lands on another branch" \
+  --seed 42 --height 512 --width 512 --num-frames 81 \
+  --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
+  --vae-use-tiling --output vace_flf2v_output.mp4
+```
+
+### Inpainting
+
+Create a mask matching the original VACE example: a 160-pixel-wide white
+vertical stripe marks the region to regenerate.
+
+```bash
+python - <<'PY'
+from PIL import Image
+
+mask = Image.new("L", (832, 480), 0)
+mask.paste(255, (336, 0, 496, 480))
+mask.save("vace_center_mask.png")
+PY
+```
+
+```bash
+python image_to_video.py \
+  --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
+  --image astronaut.jpg \
+  --mask-image vace_center_mask.png \
+  --prompt "Shrek, the ogre, walks out of a building in a happy mood" \
+  --seed 42 --height 480 --width 832 --num-frames 81 \
+  --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
+  --vae-use-tiling --output vace_inpaint_output.mp4
+```
+
+### Reference-to-Video (R2V)
+
+Repeat `--reference-image` to provide more than one reference image.
+
+```bash
+python image_to_video.py \
+  --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
+  --reference-image astronaut.jpg \
+  --prompt "Camera slowly zooms out from the character walking in a garden" \
+  --seed 42 --height 480 --width 832 --num-frames 81 \
+  --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
+  --vae-use-tiling --output vace_r2v_output.mp4
+```
+
+The VACE T2V command is documented in the shared
+[`text_to_video.py`](../text_to_video/text_to_video.md#wan21-vace-t2v) example.
+
 For Wan2.2 LightX2V-converted local Diffusers directories and related LoRA
 assets, see the [LoRA guide](../../../docs/user_guide/diffusion/lora.md#wan22-lightx2v-offline-assembly).

examples/offline_inference/image_to_video/image_to_video.py

@@ -2,13 +2,14 @@
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
 
 """
-Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, or HunyuanVideo-1.5.
+Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE.
 
 Supports:
 - Wan2.2-I2V-A14B-Diffusers: MoE model with CLIP image encoder
 - Wan2.2-TI2V-5B-Diffusers: Unified T2V+I2V model (dense 5B)
 - LTX2 image-to-video pipeline
 - HunyuanVideo-1.5 I2V: SigLIP + VAE dual image conditioning
+- Wan2.1 VACE: first/last-frame, inpainting, and reference conditioning
 
 Usage:
     # Wan I2V-A14B (MoE)
@@ -53,7 +54,11 @@ from vllm_omni.diffusion.data import DiffusionParallelConfig
 from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
 from vllm_omni.entrypoints.omni import Omni
 from vllm_omni.inputs.data import OmniDiffusionSamplingParams
-from vllm_omni.model_extras import get_extra_body_params, get_model_class_name
+from vllm_omni.model_extras import (
+    build_image_to_video_prompt,
+    get_extra_body_params,
+    get_model_class_name,
+)
 from vllm_omni.outputs import OmniRequestOutput
 from vllm_omni.platforms import current_omni_platform
 
@@ -73,18 +78,28 @@ parse_profiler_config = functools.partial(parse_json_object, flag_name="--profil
 
 
 def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(description="Generate a video from an image (Wan2.2, LTX2, HunyuanVideo-1.5).")
+    parser = argparse.ArgumentParser(
+        description="Generate a video from one or more images (Wan2.2, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE)."
+    )
     parser.add_argument(
         "--model",
         default="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
-        help="Diffusers I2V model ID or local path (Wan2.2 or HunyuanVideo-1.5).",
+        help="Diffusers I2V model ID or local path (Wan2.2, HunyuanVideo-1.5, or Wan2.1 VACE).",
     )
     parser.add_argument(
         "--model-class-name",
         default=None,
         help="Override model class name (e.g., LTX2ImageToVideoPipeline).",
     )
-    parser.add_argument("--image", required=True, help="Path to input image.")
+    parser.add_argument("--image", help="Path to the first-frame or source image.")
+    parser.add_argument("--last-image", help="Path to a last-frame condition (used by models such as VACE).")
+    parser.add_argument("--mask-image", help="Path to an inpainting mask (used by models such as VACE).")
+    parser.add_argument(
+        "--reference-image",
+        action="append",
+        default=None,
+        help="Path to a reference image. Repeat to provide multiple references.",
+    )
     parser.add_argument("--prompt", default="", help="Text prompt describing the desired motion.")
     parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
     parser.add_argument("--seed", type=int, default=42, help="Random seed.")
@@ -302,8 +317,15 @@ def main():
     if model_class_name is None and is_ltx2:
         model_class_name = "LTX2ImageToVideoPipeline"
 
-    # Load input image
-    image = PIL.Image.open(args.image).convert("RGB")
+    image = PIL.Image.open(args.image).convert("RGB") if args.image else None
+    last_image = PIL.Image.open(args.last_image).convert("RGB") if args.last_image else None
+    mask_image = PIL.Image.open(args.mask_image).convert("L") if args.mask_image else None
+    reference_images = (
+        [PIL.Image.open(path).convert("RGB") for path in args.reference_image] if args.reference_image else None
+    )
+    dimension_image = image or last_image or (reference_images[0] if reference_images else None)
+    if dimension_image is None:
+        raise ValueError("Provide --image, --last-image, or at least one --reference-image.")
 
     # Per-model generation defaults, applied only when the matching flag is omitted.
     # Cosmos3 would otherwise silently inherit the Wan2.2 defaults (wrong size/steps/shift).
@@ -333,12 +355,21 @@ def main():
     height = args.height
     width = args.width
     if height is None or width is None:
-        calc_height, calc_width = calculate_dimensions(image, max_area=d_max_area, mod_value=d_mod)
+        calc_height, calc_width = calculate_dimensions(dimension_image, max_area=d_max_area, mod_value=d_mod)
         height = height or calc_height
         width = width or calc_width
 
-    # Resize image to target dimensions
-    image = image.resize((width, height), PIL.Image.Resampling.LANCZOS)
+    media_inputs: dict[str, Any] = {}
+    if image is not None:
+        media_inputs["image"] = image.resize((width, height), PIL.Image.Resampling.LANCZOS)
+    if last_image is not None:
+        media_inputs["last_image"] = last_image.resize((width, height), PIL.Image.Resampling.LANCZOS)
+    if mask_image is not None:
+        media_inputs["mask"] = mask_image.resize((width, height), PIL.Image.Resampling.NEAREST)
+    if reference_images is not None:
+        media_inputs["reference_images"] = [
+            reference.resize((width, height), PIL.Image.Resampling.LANCZOS) for reference in reference_images
+        ]
 
     # Configure cache based on backend type
     cache_config = None
@@ -409,8 +440,8 @@ def main():
     if args.extra_body and "guardrails" in args.extra_body:
         omni_kwargs["model_config"] = {"guardrails": bool(args.extra_body["guardrails"])}
     omni = Omni(**omni_kwargs)
-    resolved_model_class_name = get_model_class_name(omni)
-    declared_extra_body_params = get_extra_body_params(resolved_model_class_name)
+    model_class_name = get_model_class_name(omni) or model_class_name
+    declared_extra_body_params = get_extra_body_params(model_class_name)
 
     if profiler_enabled:
         print("[Profiler] Starting profiling...")
@@ -434,6 +465,15 @@ def main():
     print(f"  Video size: {args.width}x{args.height}")
     print(f"{'=' * 60}\n")
 
+    prompt_dict = build_image_to_video_prompt(
+        model_class_name=model_class_name,
+        prompt=args.prompt,
+        negative_prompt=args.negative_prompt,
+        media_inputs=media_inputs,
+        height=height,
+        width=width,
+        num_frames=num_frames,
+    )
     sampling_params = OmniDiffusionSamplingParams(
         height=height,
         width=width,
@@ -461,14 +501,7 @@ def main():
 
     generation_start = time.perf_counter()
     # omni.generate() returns Generator[OmniRequestOutput, None, None]
-    frames = omni.generate(
-        {
-            "prompt": args.prompt,
-            "negative_prompt": args.negative_prompt,
-            "multi_modal_data": {"image": image},
-        },
-        sampling_params,
-    )
+    frames = omni.generate(prompt_dict, sampling_params)
     generation_end = time.perf_counter()
     generation_time = generation_end - generation_start
 

examples/offline_inference/text_to_video/text_to_video.md

@@ -6,6 +6,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
 
 | Model | Default Resolution | Default Frames | Default Steps | Guidance | VRAM (BF16) |
 |---|---|---|---|---|---|
+| `Wan-AI/Wan2.1-VACE-1.3B-diffusers` | 480x832 | 81 | 30 | 5.0 | ~20 GiB (RTX 5090, VAE tiling) |
 | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB |
 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
@@ -31,6 +32,28 @@ python text_to_video.py \
   --output t2v_out.mp4
 ```
 
+### Wan2.1 VACE (T2V)
+
+VACE text-to-video uses this shared entrypoint. Conditional VACE tasks use
+the shared [`image_to_video.py`](../image_to_video/README.md#wan21-vace-conditional-tasks)
+entrypoint, which constructs the pipeline-native conditioning data from the
+provided media inputs. No explicit mode parameter is required.
+
+```bash
+python text_to_video.py \
+  --model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
+  --prompt "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves." \
+  --seed 0 \
+  --height 480 \
+  --width 832 \
+  --num-frames 81 \
+  --num-inference-steps 30 \
+  --guidance-scale 5.0 \
+  --flow-shift 5.0 \
+  --vae-use-tiling \
+  --output vace_t2v_output.mp4
+```
+
 LTX2 example:
 
 ```bash

examples/offline_inference/text_to_video/text_to_video.py

@@ -19,6 +19,15 @@ from vllm_omni.outputs import OmniRequestOutput
 from vllm_omni.platforms import current_omni_platform
 
 _MODEL_PRESETS = {
+    "vace": {
+        "height": 480,
+        "width": 832,
+        "num_frames": 81,
+        "num_inference_steps": 30,
+        "guidance_scale": 5.0,
+        "fps": 16,
+        "output": "vace_t2v_output.mp4",
+    },
     "wan": {
         "height": 720,
         "width": 1280,
@@ -61,6 +70,8 @@ _MODEL_PRESETS = {
 
 def _detect_preset(model: str) -> dict:
     model_lower = model.lower()
+    if "vace" in model_lower:
+        return _MODEL_PRESETS["vace"]
     if "cosmos" in model_lower:
         return _MODEL_PRESETS["cosmos"]
     if "hunyuan" in model_lower:
@@ -381,7 +392,7 @@ def main():
         omni_kwargs["model_config"] = {"guardrails": bool(args.extra_body["guardrails"])}
 
     omni = Omni(**omni_kwargs)
-    model_class_name = get_model_class_name(omni)
+    model_class_name = get_model_class_name(omni) or model_class_name
     declared_extra_body_params = get_extra_body_params(model_class_name)
 
     if profiler_enabled:
@@ -419,7 +430,6 @@ def main():
         sampling_kwargs["guidance_scale_2"] = args.guidance_scale_high
 
     sampling_params = OmniDiffusionSamplingParams(**sampling_kwargs)
-
     # Route model-specific knobs through extra_body, filtered against the model's
     # declared extra_body_params. Models without a declaration only forward explicit
     # --extra-body JSON (preserving the generic flags' legacy behavior).

examples/offline_inference/vace/vace_video_generation.md

@@ -1,148 +0,0 @@
-# VACE Video Generation
-
-Generate videos from text prompts, images, or video conditions using vLLM-Omni's VACE diffusion pipeline.
-
-- `vace_video_generation.py`: command-line script for multi-mode video generation with advanced options.
-
-## Table of Contents
-
-- [Overview](#overview)
-- [Quick Start](#quick-start)
-- [Key Arguments](#key-arguments)
-- [More CLI Examples](#more-cli-examples)
-
-## Overview
-
-[VACE](https://github.com/ali-vilab/VACE) (Video All-in-one Creation Engine) supports multiple video tasks through a single unified model, including text-to-video, image-to-video, first-last-frame interpolation, inpainting, and reference image-guided generation.
-
-### Supported Models
-
-| Model | Architecture | Peak VRAM (GiB) * | Model Weights (GiB) | HuggingFace |
-|-------|-------------|-------------------|---------------------|-------------|
-| Wan2.1-VACE (1.3B) | Wan2.1 | TBD | ~10 | [Wan-AI/Wan2.1-VACE-1.3B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers) |
-| Wan2.1-VACE (14B) | Wan2.1 | TBD | ~38 | [Wan-AI/Wan2.1-VACE-14B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) |
-
-!!! info
-*Peak VRAM: based on basic single-card usage, 480x832 resolution, 81 frames, 30 inference steps, without any acceleration/optimization features.
-
-Default model: `Wan-AI/Wan2.1-VACE-14B-diffusers`
-
-## Quick Start
-
-### Python API
-
-Text-to-video generation:
-
-```python
-import torch
-
-from vllm_omni.entrypoints.omni import Omni
-from vllm_omni.inputs.data import OmniDiffusionSamplingParams
-
-if __name__ == "__main__":
-    omni = Omni(model="Wan-AI/Wan2.1-VACE-1.3B-diffusers")
-    prompt = "A sleek robot stands in a vast warehouse filled with boxes"
-    outputs = omni.generate(
-        prompt,
-        OmniDiffusionSamplingParams(
-            height=480,
-            width=832,
-            num_frames=81,
-            num_inference_steps=30,
-            guidance_scale=5.0,
-        ),
-    )
-    video = outputs[0].images
-
-    from diffusers.utils import export_to_video
-    export_to_video(list(video[0]), "t2v_output.mp4", fps=16)
-    omni.close()
-```
-
-### Local CLI Usage
-
-```bash
-python vace_video_generation.py \
-  --mode t2v \
-  --prompt "A sleek robot stands in a vast warehouse filled with boxes" \
-  --height 480 --width 832 --num-frames 81 \
-  --output t2v_output.mp4
-```
-
-## Key Arguments
-
-| Argument | Type | Default | Description |
-| -------- | ---- | ------- | ----------- |
-| `--mode` | str | `"t2v"` | VACE task mode: `t2v`, `i2v`, `v2lf`, `flf2v`, `inpaint`, `r2v` |
-| `--model` | str | `"Wan-AI/Wan2.1-VACE-14B-diffusers"` | Model ID or local path |
-| `--prompt` | str | `"A cat walking in a garden"` | Text description of desired video |
-| `--negative-prompt` | str | `""` | Negative prompt for classifier-free guidance |
-| `--image` | str | `None` | Input image path (for I2V, R2V, FLF2V, inpaint modes) |
-| `--last-image` | str | `None` | Last frame image path (for FLF2V mode) |
-| `--height` | int | `480` | Output video height in pixels (should be a multiple of 16) |
-| `--width` | int | `832` | Output video width in pixels (should be a multiple of 16) |
-| `--num-frames` | int | `81` | Number of video frames to generate |
-| `--num-inference-steps` | int | `30` | Number of denoising steps (more steps = higher quality, slower) |
-| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale |
-| `--flow-shift` | float | `5.0` | Scheduler flow shift parameter |
-| `--seed` | int | `42` | Random seed for deterministic sampling |
-| `--fps` | int | `16` | Frames per second for the saved MP4 |
-| `--output` | str | `"vace_output.mp4"` | Path to save the generated video |
-| `--vae-use-tiling` | flag | on | Enable VAE tiling for memory optimization |
-| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel |
-| `--ulysses-degree` | int | `1` | Ulysses sequence parallel degree for multi-GPU inference |
-| `--ring-degree` | int | `1` | Ring sequence parallel degree for hybrid Ulysses + Ring inference |
-| `--tensor-parallel-size` | int | — | Tensor parallel size |
-| `--enforce-eager` | flag | off | Disable torch.compile |
-
-> If you encounter OOM errors, try `--vae-use-tiling` or multi-GPU parallelism options (`--ulysses-degree`, `--cfg-parallel-size`).
-
-## More CLI Examples
-
-### Image-to-Video (I2V)
-
-First frame is kept, remaining frames are generated:
-
-```bash
-python vace_video_generation.py \
-  --mode i2v \
-  --image astronaut.jpg \
-  --prompt "An astronaut emerging from a cracked egg on the moon" \
-  --height 480 --width 832 --num-frames 81 \
-  --output i2v_output.mp4
-```
-
-### First-Last-Frame Interpolation (FLF2V)
-
-```bash
-python vace_video_generation.py \
-  --mode flf2v \
-  --image first_frame.jpg --last-image last_frame.jpg \
-  --prompt "A bird takes off from a branch and lands on another" \
-  --height 512 --width 512 --num-frames 81 \
-  --output flf2v_output.mp4
-```
-
-### Inpainting
-
-Center vertical stripe is masked and regenerated:
-
-```bash
-python vace_video_generation.py \
-  --mode inpaint \
-  --image scene.jpg \
-  --prompt "Shrek walks out of a building" \
-  --height 480 --width 832 --num-frames 81 \
-  --output inpaint_output.mp4
-```
-
-### Reference Image-guided (R2V)
-
-```bash
-python vace_video_generation.py \
-  --mode r2v \
-  --image reference.jpg \
-  --prompt "Camera slowly zooms out from the character" \
-  --height 480 --width 832 --num-frames 81 \
-  --output r2v_output.mp4
-```

examples/offline_inference/vace/vace_video_generation.py

@@ -1,209 +0,0 @@
-# SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
-
-"""VACE video generation example.
-
-VACE (Video All-in-one Creation Engine) supports multiple video tasks:
-  - T2V:        Text-to-Video
-  - I2V:        Image-to-Video (first frame conditioning)
-  - V2LF:       Video-to-Last-Frame
-  - FLF2V:      First-Last-Frame interpolation
-  - Inpainting:  Masked region generation
-  - R2V:        Reference image-guided generation
-
-Usage examples:
-  # T2V (text-to-video)
-  python vace_video_generation.py --mode t2v --prompt "A robot in a warehouse"
-
-  # I2V (image-to-video, first frame kept)
-  python vace_video_generation.py --mode i2v --image input.jpg --prompt "..."
-
-  # FLF2V (first-last frame interpolation)
-  python vace_video_generation.py --mode flf2v --image first.jpg --last-image last.jpg
-
-  # R2V (reference image guided)
-  python vace_video_generation.py --mode r2v --image ref.jpg --prompt "..."
-"""
-
-import argparse
-import time
-from pathlib import Path
-
-import numpy as np
-import PIL.Image
-import torch
-
-from vllm_omni.diffusion.data import DiffusionParallelConfig
-from vllm_omni.entrypoints.omni import Omni
-from vllm_omni.inputs.data import OmniDiffusionSamplingParams
-from vllm_omni.platforms import current_omni_platform
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(description="VACE video generation.")
-    parser.add_argument(
-        "--model",
-        default="Wan-AI/Wan2.1-VACE-14B-diffusers",
-        help="VACE model ID or local path.",
-    )
-    parser.add_argument(
-        "--mode",
-        default="t2v",
-        choices=["t2v", "i2v", "v2lf", "flf2v", "inpaint", "r2v"],
-        help="Generation mode.",
-    )
-    parser.add_argument("--prompt", default="A cat walking in a garden", help="Text prompt.")
-    parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
-    parser.add_argument("--image", type=str, default=None, help="Input image path (for I2V, R2V, FLF2V, inpaint).")
-    parser.add_argument("--last-image", type=str, default=None, help="Last frame image path (for FLF2V).")
-    parser.add_argument("--video-dir", type=str, default=None, help="Directory of video frames (for inpaint).")
-    parser.add_argument("--seed", type=int, default=42, help="Random seed.")
-    parser.add_argument("--guidance-scale", type=float, default=5.0, help="CFG guidance scale.")
-    parser.add_argument("--height", type=int, default=480, help="Video height.")
-    parser.add_argument("--width", type=int, default=832, help="Video width.")
-    parser.add_argument("--num-frames", type=int, default=81, help="Number of frames.")
-    parser.add_argument("--num-inference-steps", type=int, default=30, help="Sampling steps.")
-    parser.add_argument("--flow-shift", type=float, default=5.0, help="Scheduler flow_shift.")
-    parser.add_argument("--output", type=str, default="vace_output.mp4", help="Output video path.")
-    parser.add_argument("--fps", type=int, default=16, help="Output video FPS.")
-    parser.add_argument("--vae-use-tiling", action="store_true", default=True, help="Enable VAE tiling.")
-    parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
-    parser.add_argument("--ulysses-degree", type=int, default=1, help="Ulysses SP degree.")
-    parser.add_argument("--ring-degree", type=int, default=1, help="Ring attention degree.")
-    parser.add_argument("--cfg-parallel-size", type=int, default=1, choices=[1, 2], help="CFG parallel size.")
-    return parser.parse_args()
-
-
-def build_prompts(args):
-    """Build prompt dict with multi_modal_data based on mode."""
-    h, w, nf = args.height, args.width, args.num_frames
-
-    gray = PIL.Image.new("RGB", (w, h), (128, 128, 128))
-    mask_black = PIL.Image.new("L", (w, h), 0)
-    mask_white = PIL.Image.new("L", (w, h), 255)
-
-    prompt_data = {
-        "prompt": args.prompt,
-        "negative_prompt": args.negative_prompt,
-    }
-
-    if args.mode == "t2v":
-        return prompt_data
-
-    if args.mode == "r2v":
-        assert args.image, "--image required for R2V mode"
-        ref_img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
-        prompt_data["multi_modal_data"] = {"reference_images": [ref_img]}
-        return prompt_data
-
-    if args.mode == "i2v":
-        assert args.image, "--image required for I2V mode"
-        img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
-        prompt_data["multi_modal_data"] = {
-            "video": [img] + [gray] * (nf - 1),
-            "mask": [mask_black] + [mask_white] * (nf - 1),
-        }
-        return prompt_data
-
-    if args.mode == "v2lf":
-        assert args.image, "--image required for V2LF mode"
-        img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
-        prompt_data["multi_modal_data"] = {
-            "video": [gray] * (nf - 1) + [img],
-            "mask": [mask_white] * (nf - 1) + [mask_black],
-        }
-        return prompt_data
-
-    if args.mode == "flf2v":
-        assert args.image and args.last_image, "--image and --last-image required for FLF2V"
-        first = PIL.Image.open(args.image).convert("RGB").resize((w, h))
-        last = PIL.Image.open(args.last_image).convert("RGB").resize((w, h))
-        prompt_data["multi_modal_data"] = {
-            "video": [first] + [gray] * (nf - 2) + [last],
-            "mask": [mask_black] + [mask_white] * (nf - 2) + [mask_black],
-        }
-        return prompt_data
-
-    if args.mode == "inpaint":
-        assert args.image, "--image required for inpaint mode"
-        img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
-        d = 80
-        frames, masks = [], []
-        for _ in range(nf):
-            base = np.array(img).copy()
-            mask = PIL.Image.new("L", (w, h), 0)
-            stripe = PIL.Image.new("L", (2 * d, h), 255)
-            mask.paste(stripe, (w // 2 - d, 0))
-            base[np.array(mask) > 128] = 128
-            frames.append(PIL.Image.fromarray(base))
-            masks.append(mask)
-        prompt_data["multi_modal_data"] = {"video": frames, "mask": masks}
-        return prompt_data
-
-    raise ValueError(f"Unknown mode: {args.mode}")
-
-
-def main():
-    args = parse_args()
-    generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
-
-    parallel_config = DiffusionParallelConfig(
-        ulysses_degree=args.ulysses_degree,
-        ring_degree=args.ring_degree,
-        cfg_parallel_size=args.cfg_parallel_size,
-    )
-
-    omni = Omni(
-        model=args.model,
-        vae_use_tiling=args.vae_use_tiling,
-        flow_shift=args.flow_shift,
-        enforce_eager=args.enforce_eager,
-        parallel_config=parallel_config,
-    )
-
-    prompt_data = build_prompts(args)
-
-    print(f"\n{'=' * 60}")
-    print(f"VACE {args.mode.upper()} Generation")
-    print(f"  Model: {args.model}")
-    print(f"  Size: {args.width}x{args.height}, {args.num_frames} frames, {args.num_inference_steps} steps")
-    print(f"{'=' * 60}\n")
-
-    start = time.perf_counter()
-    outputs = omni.generate(
-        prompt_data,
-        OmniDiffusionSamplingParams(
-            height=args.height,
-            width=args.width,
-            num_frames=args.num_frames,
-            num_inference_steps=args.num_inference_steps,
-            guidance_scale=args.guidance_scale,
-            generator=generator,
-        ),
-    )
-    elapsed = time.perf_counter() - start
-
-    video = outputs[0].images
-    if isinstance(video, list):
-        video = video[0]
-    if isinstance(video, torch.Tensor):
-        video = video.cpu().numpy()
-    if video.ndim == 5:
-        video = video[0]
-    print(f"Output shape: {video.shape}, Time: {elapsed:.1f}s")
-
-    output_path = Path(args.output)
-    output_path.parent.mkdir(parents=True, exist_ok=True)
-
-    from diffusers.utils import export_to_video
-
-    if np.issubdtype(video.dtype, np.integer):
-        video = video.astype(np.float32) / 255.0
-    export_to_video(list(video), str(output_path), fps=args.fps)
-    print(f"Saved to {output_path}")
-
-    omni.close()
-
-
-if __name__ == "__main__":
-    main()

recipes/README.md

@@ -63,6 +63,7 @@ recipes/
 | [`Tencent/HunyuanImage-3.0-Instruct.md`](./Tencent/HunyuanImage-3.0-Instruct.md) | DiT-only text-to-image serving and benchmark, including ModelOpt mixed FP8/NVFP4 | 4x H100/H800 80GB / 2x B200 |
 | [`Wan-AI/Wan2.2-I2V.md`](./Wan-AI/Wan2.2-I2V.md) | Image-to-video serving (Wan2.2 14B) | 8x Ascend NPU (A2/A3) |
 | [`Wan-AI/Wan2.2-S2V.md`](./Wan-AI/Wan2.2-S2V.md) | Speech-to-video serving (Wan2.2 14B) | 2x A100/H100 80GB |
+| [`Wan-AI/Wan2.1-VACE.md`](./Wan-AI/Wan2.1-VACE.md) | Unified T2V, I2V, V2LF, FLF2V, inpaint, and R2V | 1x RTX 5090 (1.3B) / 1x L40S 48GB with layerwise offload (14B) |
 | [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB |
 | [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB |
 | [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB |

recipes/Wan-AI/Wan2.1-VACE.md

@@ -0,0 +1,147 @@
+# Wan2.1 VACE Unified Video Generation
+
+## Summary
+
+- Vendor: Wan-AI
+- Models: `Wan2.1-VACE-1.3B-diffusers`, `Wan2.1-VACE-14B-diffusers`
+- Tasks: T2V, I2V, V2LF, FLF2V, inpainting, and R2V
+- Mode: offline inference through the shared task examples
+
+The shared examples construct VACE's pipeline-native conditioning data from the
+provided media inputs. They do not require a mode parameter or a model-specific
+example script.
+
+## References
+
+- [VACE project](https://github.com/ali-vilab/VACE)
+- [Hugging Face 14B checkpoint](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers)
+- [ModelScope 14B checkpoint](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B-diffusers)
+- [Shared text-to-video example](../../examples/offline_inference/text_to_video/text_to_video.md)
+- [Shared image-to-video example](../../examples/offline_inference/image_to_video/README.md)
+- [Migration RFC #4539](https://github.com/vllm-project/vllm-omni/issues/4539)
+
+## Hardware Support
+
+The 1.3B checkpoint was validated on an RTX 5090; the 81-frame T2V command below
+used approximately 20 GiB of peak reserved GPU memory with VAE tiling. The 14B
+checkpoint requires approximately 70 GiB of disk space and was validated on a
+single 48 GiB L40S with VAE tiling and layerwise offload. Start with the smoke
+configuration before attempting an 81-frame run.
+
+## Setup
+
+The public examples use the Hugging Face model ID by default:
+
+```bash
+MODEL="${MODEL:-Wan-AI/Wan2.1-VACE-14B-diffusers}"
+ASSET_DIR="${ASSET_DIR:-$HOME/datasets/vace}"
+OUT="${OUT:-$PWD/vace-results/14b}"
+mkdir -p "$ASSET_DIR" "$OUT"
+```
+
+For environments where Hugging Face downloads are unavailable, download the
+same checkpoint from ModelScope and point `MODEL` at the local directory:
+
+```bash
+MODEL="$HOME/models/Wan2.1-VACE-14B-diffusers"
+modelscope download --model Wan-AI/Wan2.1-VACE-14B-diffusers --local_dir "$MODEL"
+```
+
+Download the public conditioning images and create a mask matching the output
+size. White pixels are regenerated and black pixels are preserved.
+
+```bash
+wget -O "$ASSET_DIR/astronaut.jpg" \
+  https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg
+wget -O "$ASSET_DIR/vace_first_frame.png" \
+  https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png
+wget -O "$ASSET_DIR/vace_last_frame.png" \
+  https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png
+
+python - "$ASSET_DIR/vace_center_mask.png" <<'PY'
+import sys
+from PIL import Image
+
+mask = Image.new("L", (832, 480), 0)
+mask.paste(255, (336, 0, 496, 480))
+mask.save(sys.argv[1])
+PY
+
+ASTRONAUT="$ASSET_DIR/astronaut.jpg"
+FIRST="$ASSET_DIR/vace_first_frame.png"
+LAST="$ASSET_DIR/vace_last_frame.png"
+MASK="$ASSET_DIR/vace_center_mask.png"
+```
+
+## Commands
+
+The commands below reproduce the original VACE examples with `81 frames / 30
+steps`. Most tasks use `480x832`; FLF2V uses `512x512`. For a smoke test,
+replace the frame and step counts with `5 frames / 2 steps`.
+
+```bash
+# T2V
+python examples/offline_inference/text_to_video/text_to_video.py \
+  --model "$MODEL" \
+  --prompt "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves." \
+  --seed 0 \
+  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
+  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
+  --output "$OUT/t2v.mp4"
+
+# I2V
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model "$MODEL" --image "$ASTRONAUT" \
+  --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
+  --seed 42 \
+  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
+  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
+  --output "$OUT/i2v.mp4"
+
+# V2LF
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model "$MODEL" --last-image "$ASTRONAUT" \
+  --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
+  --seed 42 \
+  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
+  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
+  --output "$OUT/v2lf.mp4"
+
+# FLF2V
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model "$MODEL" --image "$FIRST" --last-image "$LAST" \
+  --prompt "CG animation style, a small blue bird takes off from a branch and lands on another branch" \
+  --seed 42 --height 512 --width 512 --num-frames 81 --num-inference-steps 30 \
+  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
+  --output "$OUT/flf2v.mp4"
+
+# Inpaint
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model "$MODEL" --image "$ASTRONAUT" --mask-image "$MASK" \
+  --prompt "Shrek, the ogre, walks out of a building in a happy mood" \
+  --seed 42 \
+  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
+  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
+  --output "$OUT/inpaint.mp4"
+
+# R2V; repeat --reference-image to add references
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model "$MODEL" --reference-image "$ASTRONAUT" \
+  --prompt "Camera slowly zooms out from the character walking in a garden" \
+  --seed 42 \
+  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
+  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
+  --output "$OUT/r2v.mp4"
+```
+
+## Verification
+
+```bash
+for video in "$OUT"/*.mp4; do
+  ffprobe -v error -select_streams v:0 \
+    -show_entries stream=codec_name,width,height,nb_frames,r_frame_rate,duration \
+    -of default=noprint_wrappers=1 "$video"
+done
+```
+
+For the 1.3B checkpoint, `--enable-layerwise-offload` is generally unnecessary.

tests/examples/offline_inference/test_vace_video_generation.py

@@ -1,11 +1,9 @@
-"""
-Offline inference tests: vace video generation.
-See examples/offline_inference/vace/vace_video_generation.md
-"""
+"""VACE coverage through the shared text-to-video and image-to-video examples."""
 
 from pathlib import Path
 
 import pytest
+from PIL import Image
 
 from tests.examples.helpers import EXAMPLES, ExampleRunner, ReadmeSnippet
 from tests.helpers.assertions import assert_video_valid
@@ -14,22 +12,83 @@ from tests.helpers.mark import hardware_marks
 pytestmark = [
     pytest.mark.usefixtures("clean_gpu_memory_between_tests"),
     pytest.mark.full_model,
+    pytest.mark.diffusion,
     pytest.mark.example,
     *hardware_marks(res={"cuda": "H100"}),
 ]
 
-VACE_SCRIPT = EXAMPLES / "offline_inference" / "vace" / "vace_video_generation.py"
-README_PATH = VACE_SCRIPT.with_name("vace_video_generation.md")
+TEXT_TO_VIDEO_README = EXAMPLES / "offline_inference" / "text_to_video" / "text_to_video.md"
+IMAGE_TO_VIDEO_README = EXAMPLES / "offline_inference" / "image_to_video" / "README.md"
 EXAMPLE_OUTPUT_SUBFOLDER = "example_offline_vace"
 
-README_SNIPPETS = ReadmeSnippet.extract_readme_snippets(README_PATH)
+TEXT_TO_VIDEO_SNIPPET = next(
+    snippet
+    for snippet in ReadmeSnippet.extract_readme_snippets(TEXT_TO_VIDEO_README)
+    if "vace_t2v_output.mp4" in snippet.code
+)
+CONDITIONAL_OUTPUTS = (
+    "vace_i2v_output.mp4",
+    "vace_v2lf_output.mp4",
+    "vace_flf2v_output.mp4",
+    "vace_inpaint_output.mp4",
+    "vace_r2v_output.mp4",
+)
+IMAGE_TO_VIDEO_SNIPPETS = [
+    snippet
+    for snippet in ReadmeSnippet.extract_readme_snippets(IMAGE_TO_VIDEO_README)
+    if snippet.h2_title == "Wan2.1 VACE Conditional Tasks"
+    and any(output in snippet.code for output in CONDITIONAL_OUTPUTS)
+]
 
 
-@pytest.mark.parametrize("snippet", README_SNIPPETS, ids=lambda snippet: snippet.test_id)
-def test_vace_video_generation(snippet: ReadmeSnippet, example_runner: ExampleRunner):
-    should_skip, reason = snippet.skip
-    if should_skip:
-        pytest.skip(reason)
+def _smoke_snippet(snippet: ReadmeSnippet, asset_dir: Path) -> ReadmeSnippet:
+    # Six fresh model launches share a 90-minute X2V job. Keep the documented
+    # quality settings intact while reducing only the CI executions.
+    replacements = {
+        "astronaut.jpg": str(asset_dir / "astronaut.jpg"),
+        "vace_first_frame.png": str(asset_dir / "vace_first_frame.png"),
+        "vace_last_frame.png": str(asset_dir / "vace_last_frame.png"),
+        "vace_center_mask.png": str(asset_dir / "vace_center_mask.png"),
+        "--num-frames 81": "--num-frames 5",
+        "--num-inference-steps 30": "--num-inference-steps 2",
+    }
+    code = snippet.code
+    for source, target in replacements.items():
+        code = code.replace(source, target)
+    return snippet._replace(code=code)
+
+
+@pytest.fixture
+def vace_asset_dir(tmp_path: Path) -> Path:
+    asset_dir = tmp_path / "vace_assets"
+    asset_dir.mkdir(parents=True, exist_ok=True)
+    Image.new("RGB", (832, 480), (180, 180, 180)).save(asset_dir / "astronaut.jpg")
+    Image.new("RGB", (512, 512), (80, 140, 220)).save(asset_dir / "vace_first_frame.png")
+    Image.new("RGB", (512, 512), (220, 140, 80)).save(asset_dir / "vace_last_frame.png")
+    mask = Image.new("L", (832, 480), 0)
+    mask.paste(255, (336, 0, 496, 480))
+    mask.save(asset_dir / "vace_center_mask.png")
+    return asset_dir
+
+
+def test_vace_text_to_video_shared_example(example_runner: ExampleRunner) -> None:
+    snippet = TEXT_TO_VIDEO_SNIPPET._replace(
+        code=TEXT_TO_VIDEO_SNIPPET.code.replace("--num-frames 81", "--num-frames 5").replace(
+            "--num-inference-steps 30", "--num-inference-steps 2"
+        )
+    )
+    result = example_runner.run(snippet, output_subfolder=Path(EXAMPLE_OUTPUT_SUBFOLDER))
+    for asset in result.assets:
+        assert_video_valid(asset)
+
+
+@pytest.mark.parametrize("snippet", IMAGE_TO_VIDEO_SNIPPETS, ids=lambda snippet: snippet.test_id)
+def test_vace_conditional_shared_examples(
+    snippet: ReadmeSnippet,
+    example_runner: ExampleRunner,
+    vace_asset_dir: Path,
+) -> None:
+    snippet = _smoke_snippet(snippet, vace_asset_dir)
     result = example_runner.run(snippet, output_subfolder=Path(EXAMPLE_OUTPUT_SUBFOLDER))
     for asset in result.assets:
         assert_video_valid(asset)

tests/model_extras/test_model_extras.py

@@ -10,6 +10,7 @@ from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
 from vllm_omni.inputs.data import OmniDiffusionSamplingParams
 from vllm_omni.model_extras import (
     build_image_to_image_prompt,
+    build_image_to_video_prompt,
     build_text_to_image_prompt,
     get_extra_body_params,
     get_extra_output_params,
@@ -176,6 +177,14 @@ def test_helios_extra_registry_declares_request_and_response_params() -> None:
         assert should_init_extra_args_for_non_diffusion_stages(cls) is False
 
 
+@pytest.mark.core_model
+@pytest.mark.cpu
+def test_vace_extra_registry_has_no_pipeline_params() -> None:
+    assert get_extra_body_params("WanVACEPipeline") == frozenset()
+    assert get_extra_output_params("WanVACEPipeline") == frozenset()
+    assert should_init_extra_args_for_non_diffusion_stages("WanVACEPipeline") is False
+
+
 @pytest.mark.core_model
 @pytest.mark.cpu
 def test_unknown_pipeline_has_empty_extra_registry() -> None:
@@ -253,6 +262,58 @@ def test_unknown_pipeline_uses_default_image_to_image_prompt() -> None:
     }
 
 
+def _build_vace_prompt(media_inputs: dict[str, object], *, num_frames: int = 5) -> dict:
+    return build_image_to_video_prompt(
+        "WanVACEPipeline",
+        prompt="a bird flying",
+        negative_prompt=None,
+        media_inputs=media_inputs,
+        height=16,
+        width=320,
+        num_frames=num_frames,
+    )
+
+
+@pytest.mark.core_model
+@pytest.mark.cpu
+@pytest.mark.parametrize(
+    "media_inputs",
+    [
+        {"image": Image.new("RGB", (320, 16), "red")},
+        {"last_image": Image.new("RGB", (320, 16), "blue")},
+        {
+            "image": Image.new("RGB", (320, 16), "red"),
+            "last_image": Image.new("RGB", (320, 16), "blue"),
+        },
+        {"image": Image.new("RGB", (320, 16), "red"), "mask": Image.new("L", (320, 16), 0)},
+        {"reference_images": [Image.new("RGB", (64, 64), "red")]},
+    ],
+    ids=["i2v", "v2lf", "flf2v", "inpaint", "r2v"],
+)
+def test_vace_image_to_video_prompt_builder(media_inputs: dict[str, object]) -> None:
+    result = _build_vace_prompt(media_inputs)
+    mmd = result["multi_modal_data"]
+    if "reference_images" in mmd:
+        assert mmd["reference_images"] is media_inputs["reference_images"]
+    else:
+        assert len(mmd["video"]) == len(mmd["mask"]) == 5
+
+
+@pytest.mark.core_model
+@pytest.mark.cpu
+@pytest.mark.parametrize(
+    ("media_inputs", "message"),
+    [
+        ({}, "requires a conditioning media input"),
+        ({"mask": Image.new("L", (320, 16))}, "mask input requires an image"),
+        ({"control_image": Image.new("RGB", (320, 16))}, "Unsupported VACE media input"),
+    ],
+)
+def test_vace_rejects_invalid_media_combinations(media_inputs: dict[str, object], message: str) -> None:
+    with pytest.raises(ValueError, match=message):
+        _build_vace_prompt(media_inputs)
+
+
 @pytest.mark.core_model
 @pytest.mark.cpu
 def test_declared_extra_args_apply_to_existing_sampling_params() -> None:

vllm_omni/model_extras/__init__.py

@@ -3,6 +3,7 @@
 
 from vllm_omni.model_extras.registry import (
     build_image_to_image_prompt,
+    build_image_to_video_prompt,
     build_text_to_image_prompt,
     get_extra_body_params,
     get_extra_output_params,
@@ -12,6 +13,7 @@ from vllm_omni.model_extras.registry import (
 
 __all__ = [
     "build_image_to_image_prompt",
+    "build_image_to_video_prompt",
     "build_text_to_image_prompt",
     "get_extra_body_params",
     "get_extra_output_params",

vllm_omni/model_extras/registry.py

@@ -3,7 +3,7 @@
 
 from __future__ import annotations
 
-from collections.abc import Callable
+from collections.abc import Callable, Mapping
 from typing import Any
 
 from PIL import Image
@@ -38,9 +38,14 @@ from vllm_omni.model_extras.sensenova_u1 import (
     SENSENOVA_U1_EXTRA_BODY_PARAMS,
     SENSENOVA_U1_EXTRA_OUTPUT_PARAMS,
 )
+from vllm_omni.model_extras.vace import (
+    VACE_EXTRA_BODY_PARAMS,
+    VACE_EXTRA_OUTPUT_PARAMS,
+)
+from vllm_omni.model_extras.vace import (
+    build_image_to_video_prompt as build_vace_image_to_video_prompt,
+)
 
-# TODO: extend with TextToVideoPromptBuilder, ImageToVideoPromptBuilder
-# once a video model needs model-specific prompt construction (follow-up PR).
 TextToImagePromptBuilder = Callable[
     [str, str | None, int | None, int | None],
     dict[str, Any],
@@ -49,6 +54,17 @@ ImageToImagePromptBuilder = Callable[
     [str, str | None, "Image.Image | list[Image.Image]", int | None, int | None],
     dict[str, Any],
 ]
+ImageToVideoPromptBuilder = Callable[
+    [
+        str,
+        str | None,
+        "Mapping[str, Any]",
+        int | None,
+        int | None,
+        int | None,
+    ],
+    dict[str, Any],
+]
 
 
 def default_text_to_image_prompt(
@@ -79,6 +95,20 @@ def default_image_to_image_prompt(
     return result
 
 
+def default_image_to_video_prompt(
+    prompt: str,
+    negative_prompt: str | None,
+    media_inputs: Mapping[str, Any],
+    height: int | None = None,
+    width: int | None = None,
+    num_frames: int | None = None,
+) -> dict[str, Any]:
+    del height, width, num_frames
+    if set(media_inputs) != {"image"} or not isinstance(media_inputs["image"], Image.Image):
+        raise ValueError("This model only supports a single --image input in the shared image-to-video example.")
+    return default_image_to_image_prompt(prompt, negative_prompt, media_inputs["image"])
+
+
 _EXTRA_SPECS: dict[str, dict[str, Any]] = {
     "BagelPipeline": {
         "extra_body_params": BAGEL_EXTRA_BODY_PARAMS,
@@ -108,6 +138,11 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
         "extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
         "extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,
     },
+    "WanVACEPipeline": {
+        "extra_body_params": VACE_EXTRA_BODY_PARAMS,
+        "extra_output_params": VACE_EXTRA_OUTPUT_PARAMS,
+        "image_to_video_prompt_builder": build_vace_image_to_video_prompt,
+    },
 }
 
 
@@ -177,3 +212,21 @@ def build_image_to_image_prompt(
         else default_image_to_image_prompt
     )
     return builder(prompt, negative_prompt, input_image, height, width)
+
+
+def build_image_to_video_prompt(
+    model_class_name: str | None,
+    prompt: str,
+    negative_prompt: str | None,
+    media_inputs: Mapping[str, Any],
+    height: int | None = None,
+    width: int | None = None,
+    num_frames: int | None = None,
+) -> dict[str, Any]:
+    spec = _get_spec(model_class_name)
+    builder: ImageToVideoPromptBuilder = (
+        spec.get("image_to_video_prompt_builder", default_image_to_video_prompt)
+        if spec is not None
+        else default_image_to_video_prompt
+    )
+    return builder(prompt, negative_prompt, media_inputs, height, width, num_frames)

vllm_omni/model_extras/vace.py

@@ -0,0 +1,120 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+import numpy as np
+from PIL import Image
+
+# VACE conditioning is expressed through prompt media, and the pipeline does not
+# currently read any model-specific values from sampling_params.extra_args.
+VACE_EXTRA_BODY_PARAMS: frozenset[str] = frozenset()
+VACE_EXTRA_OUTPUT_PARAMS: frozenset[str] = frozenset()
+
+_MEDIA_KEYS = frozenset({"image", "last_image", "mask", "reference_images"})
+
+
+def _base_prompt(prompt: str, negative_prompt: str | None) -> dict[str, Any]:
+    result: dict[str, Any] = {"prompt": prompt}
+    if negative_prompt is not None:
+        result["negative_prompt"] = negative_prompt
+    return result
+
+
+def _require_dimensions(
+    height: int | None,
+    width: int | None,
+    num_frames: int | None,
+) -> tuple[int, int, int]:
+    if height is None or width is None or num_frames is None:
+        raise ValueError("VACE conditional inputs require height, width, and num_frames.")
+    if height <= 0 or width <= 0 or num_frames <= 0:
+        raise ValueError("VACE height, width, and num_frames must be positive.")
+    return height, width, num_frames
+
+
+def _validate_media_inputs(media_inputs: Mapping[str, Any]) -> None:
+    unknown = set(media_inputs) - _MEDIA_KEYS
+    if unknown:
+        raise ValueError(f"Unsupported VACE media input(s): {', '.join(sorted(unknown))}.")
+
+    has_references = bool(media_inputs.get("reference_images"))
+    has_frame_conditioning = any(media_inputs.get(key) is not None for key in ("image", "last_image", "mask"))
+    if has_references and has_frame_conditioning:
+        raise ValueError("VACE reference images cannot be combined with image, last_image, or mask inputs.")
+    if media_inputs.get("mask") is not None and media_inputs.get("image") is None:
+        raise ValueError("VACE mask input requires an image input.")
+    if media_inputs.get("mask") is not None and media_inputs.get("last_image") is not None:
+        raise ValueError("VACE mask and last_image inputs cannot be combined.")
+    if not has_references and not has_frame_conditioning:
+        raise ValueError("VACE image-to-video generation requires a conditioning media input.")
+
+
+def build_image_to_video_prompt(
+    prompt: str,
+    negative_prompt: str | None,
+    media_inputs: Mapping[str, Any],
+    height: int | None = None,
+    width: int | None = None,
+    num_frames: int | None = None,
+) -> dict[str, Any]:
+    """Build VACE's native video, mask, or reference-image conditioning."""
+    _validate_media_inputs(media_inputs)
+    result = _base_prompt(prompt, negative_prompt)
+
+    reference_images = media_inputs.get("reference_images")
+    if reference_images:
+        if not isinstance(reference_images, list) or not all(
+            isinstance(image, Image.Image) for image in reference_images
+        ):
+            raise ValueError("VACE reference_images must be a non-empty list of PIL images.")
+        result["multi_modal_data"] = {"reference_images": reference_images}
+        return result
+
+    height, width, num_frames = _require_dimensions(height, width, num_frames)
+    first_image = media_inputs.get("image")
+    last_image = media_inputs.get("last_image")
+    mask_image = media_inputs.get("mask")
+    for name, image in (("image", first_image), ("last_image", last_image), ("mask", mask_image)):
+        if image is not None and not isinstance(image, Image.Image):
+            raise ValueError(f"VACE {name} must be a PIL image.")
+
+    gray = Image.new("RGB", (width, height), (128, 128, 128))
+    mask_black = Image.new("L", (width, height), 0)
+    mask_white = Image.new("L", (width, height), 255)
+
+    if mask_image is not None:
+        mask = mask_image.convert("L")
+        source = np.asarray(first_image).copy()
+        source[np.asarray(mask) > 128] = 128
+        frame = Image.fromarray(source)
+        result["multi_modal_data"] = {
+            "video": [frame.copy() for _ in range(num_frames)],
+            "mask": [mask.copy() for _ in range(num_frames)],
+        }
+        return result
+
+    if first_image is not None and last_image is not None:
+        if num_frames < 2:
+            raise ValueError("VACE first/last-frame conditioning requires at least two frames.")
+        result["multi_modal_data"] = {
+            "video": [first_image] + [gray] * (num_frames - 2) + [last_image],
+            "mask": [mask_black] + [mask_white] * (num_frames - 2) + [mask_black],
+        }
+        return result
+
+    if first_image is not None:
+        result["multi_modal_data"] = {
+            "video": [first_image] + [gray] * (num_frames - 1),
+            "mask": [mask_black] + [mask_white] * (num_frames - 1),
+        }
+        return result
+
+    result["multi_modal_data"] = {
+        "video": [gray] * (num_frames - 1) + [last_image],
+        "mask": [mask_white] * (num_frames - 1) + [mask_black],
+    }
+    return result