vllm-omni/examples/offline_inference/vace/vace_video_generation.md

VACE Video Generation

Generate videos from text prompts, images, or video conditions using vLLM-Omni's VACE diffusion pipeline.

• vace_video_generation.py: command-line script for multi-mode video generation with advanced options.

Table of Contents

• Overview
• Quick Start
• Key Arguments
• More CLI Examples

Overview

VACE (Video All-in-one Creation Engine) supports multiple video tasks through a single unified model, including text-to-video, image-to-video, first-last-frame interpolation, inpainting, and reference image-guided generation.

Supported Models

Model	Architecture	Peak VRAM (GiB) *	Model Weights (GiB)	HuggingFace
Wan2.1-VACE (1.3B)	Wan2.1	TBD	~10	Wan-AI/Wan2.1-VACE-1.3B-diffusers
Wan2.1-VACE (14B)	Wan2.1	TBD	~38	Wan-AI/Wan2.1-VACE-14B-diffusers

!!! info *Peak VRAM: based on basic single-card usage, 480x832 resolution, 81 frames, 30 inference steps, without any acceleration/optimization features.

Default model: Wan-AI/Wan2.1-VACE-14B-diffusers

Quick Start

Python API

Text-to-video generation:

import torch

from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams

if __name__ == "__main__":
    omni = Omni(model="Wan-AI/Wan2.1-VACE-1.3B-diffusers")
    prompt = "A sleek robot stands in a vast warehouse filled with boxes"
    outputs = omni.generate(
        prompt,
        OmniDiffusionSamplingParams(
            height=480,
            width=832,
            num_frames=81,
            num_inference_steps=30,
            guidance_scale=5.0,
        ),
    )
    video = outputs[0].images

    from diffusers.utils import export_to_video
    export_to_video(list(video[0]), "t2v_output.mp4", fps=16)
    omni.close()

Local CLI Usage

python vace_video_generation.py \
  --mode t2v \
  --prompt "A sleek robot stands in a vast warehouse filled with boxes" \
  --height 480 --width 832 --num-frames 81 \
  --output t2v_output.mp4

Key Arguments

Argument	Type	Default	Description
--mode	str	"t2v"	VACE task mode: t2v, i2v, v2lf, flf2v, inpaint, r2v
--model	str	"Wan-AI/Wan2.1-VACE-14B-diffusers"	Model ID or local path
--prompt	str	"A cat walking in a garden"	Text description of desired video
--negative-prompt	str	""	Negative prompt for classifier-free guidance
--image	str	None	Input image path (for I2V, R2V, FLF2V, inpaint modes)
--last-image	str	None	Last frame image path (for FLF2V mode)
--height	int	480	Output video height in pixels (should be a multiple of 16)
--width	int	832	Output video width in pixels (should be a multiple of 16)
--num-frames	int	81	Number of video frames to generate
--num-inference-steps	int	30	Number of denoising steps (more steps = higher quality, slower)
--guidance-scale	float	5.0	Classifier-free guidance scale
--flow-shift	float	5.0	Scheduler flow shift parameter
--seed	int	42	Random seed for deterministic sampling
--fps	int	16	Frames per second for the saved MP4
--output	str	"vace_output.mp4"	Path to save the generated video
--vae-use-tiling	flag	on	Enable VAE tiling for memory optimization
--cfg-parallel-size	int	1	Set to 2 to enable CFG Parallel
--ulysses-degree	int	1	Ulysses sequence parallel degree for multi-GPU inference
--ring-degree	int	1	Ring sequence parallel degree for hybrid Ulysses + Ring inference
--tensor-parallel-size	int	—	Tensor parallel size
--enforce-eager	flag	off	Disable torch.compile

If you encounter OOM errors, try --vae-use-tiling or multi-GPU parallelism options (--ulysses-degree, --cfg-parallel-size).

More CLI Examples

Image-to-Video (I2V)

First frame is kept, remaining frames are generated:

python vace_video_generation.py \
  --mode i2v \
  --image astronaut.jpg \
  --prompt "An astronaut emerging from a cracked egg on the moon" \
  --height 480 --width 832 --num-frames 81 \
  --output i2v_output.mp4

First-Last-Frame Interpolation (FLF2V)

python vace_video_generation.py \
  --mode flf2v \
  --image first_frame.jpg --last-image last_frame.jpg \
  --prompt "A bird takes off from a branch and lands on another" \
  --height 512 --width 512 --num-frames 81 \
  --output flf2v_output.mp4

Inpainting

Center vertical stripe is masked and regenerated:

python vace_video_generation.py \
  --mode inpaint \
  --image scene.jpg \
  --prompt "Shrek walks out of a building" \
  --height 480 --width 832 --num-frames 81 \
  --output inpaint_output.mp4

Reference Image-guided (R2V)

python vace_video_generation.py \
  --mode r2v \
  --image reference.jpg \
  --prompt "Camera slowly zooms out from the character" \
  --height 480 --width 832 --num-frames 81 \
  --output r2v_output.mp4