anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Feature][Refactor] LTX-2.3 I2V Support, Public Examples, and Shared Pipeline Refactor (#4739)

Browse Source 
Signed-off-by: mglyn <1203789601@qq.com>
This commit is contained in:
Mu GuanLin authored and GitHub committed 2026-07-03 11:29:22 +08:00
1 parent b443504d61
commit 3023bc40c2
14 files changed
+2897 -1176
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/image_to_video/README.md
+34 -5
View File
@@ -1,7 +1,7 @@
# Image-To-Video
This shared example generates videos from images with VACE, Wan2.2, LTX2,
HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
This shared example generates videos from images with VACE, Wan2.2,
LTX-2/LTX-2.3, HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
## Local CLI Usage
@@ -49,6 +49,30 @@ python image_to_video.py \
--output i2v_output.mp4
```
### LTX-2.3
```bash
python image_to_video.py \
--model dg845/LTX-2.3-Diffusers \
--model-class-name LTX23ImageToVideoPipeline \
--image cherry_blossom.jpg \
--prompt "Cherry blossoms swaying gently in the breeze with synchronized ambient sound" \
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
--height 384 \
--width 512 \
--num-frames 25 \
--guidance-scale 4.0 \
--num-inference-steps 20 \
--frame-rate 24 \
--fps 24 \
--output ltx23_i2v_output.mp4
```
Use a Diffusers-format checkpoint such as `dg845/LTX-2.3-Diffusers`; the
upstream `Lightricks/LTX-2.3` raw safetensors repo is not directly loadable by
this pipeline. Pass `--model-class-name LTX23ImageToVideoPipeline` to select
the LTX-2.3 image-to-video pipeline.
### Cosmos3
```bash
@@ -66,15 +90,19 @@ python image_to_video.py \
Key arguments:
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V).
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V,
LTX-2/LTX-2.3, Cosmos3, or VACE).
- `--image`: Path to the first-frame or source image.
- `--last-image`: Optional last-frame condition for models such as VACE.
- `--mask-image`: Optional inpainting mask. White pixels are regenerated and black pixels are preserved.
- `--reference-image`: Optional reference image; repeat it to provide multiple references.
- `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3.
- `--prompt`: Text description of desired motion/animation.
- `--height/--width`: Output resolution (auto-calculated from image if not set). Dimensions should be multiples of 16.
- `--num-frames`: Number of frames (default: model-specific — Wan 81, LTX2 121, Cosmos3 189).
- `--height/--width`: Output resolution (auto-calculated from image if not set).
Wan dimensions should be multiples of 16; LTX dimensions should be multiples
of 32.
- `--num-frames`: Number of frames (model-specific default; LTX-style models
work best with `8k + 1`; Cosmos3 defaults to 189).
- `--guidance-scale` and `--guidance-scale-high`: CFG scale (applied to low/high-noise stages for MoE).
- `--negative-prompt`: Optional list of artifacts to suppress.
- `--boundary-ratio`: Boundary split ratio for two-stage MoE models.
@@ -82,6 +110,7 @@ Key arguments:
- `--sample-solver`: Wan2.2 sampling solver. Use `unipc` for the default multistep solver, or `euler` for Lightning/Distill checkpoints.
- `--num-inference-steps`: Number of denoising steps (default: model-specific — Wan 50, LTX2 40, Cosmos3 35).
- `--fps`: Frames per second for the saved MP4 (requires `diffusers` export_to_video).
- `--audio-sample-rate`: fallback audio sample rate for embedded audio.
- `--output`: Path to save the generated video.
- `--vae-use-slicing`: Enable VAE slicing for memory optimization.
- `--vae-use-tiling`: Enable VAE tiling for memory optimization.
examples/offline_inference/image_to_video/image_to_video.py
+37 -11
View File
@@ -2,7 +2,8 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE.
Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2/LTX-2.3,
HunyuanVideo-1.5, Cosmos3, or Wan2.1 VACE.
Supports:
- Wan2.2-I2V-A14B-Diffusers: MoE model with CLIP image encoder
@@ -27,6 +28,13 @@ Usage:
--num-frames 121 --num-inference-steps 40 --guidance-scale 4.0 \
--frame-rate 24 --fps 24 --output ltx2_i2v.mp4
# LTX-2.3 image-to-video
python image_to_video.py --model dg845/LTX-2.3-Diffusers \
--model-class-name LTX23ImageToVideoPipeline \
--image input.jpg --prompt "A cinematic dolly shot of a boat" \
--height 384 --width 512 --num-frames 25 --num-inference-steps 20 \
--frame-rate 24 --fps 24 --output ltx23_i2v.mp4
# HunyuanVideo-1.5 I2V (480p)
python image_to_video.py --model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v \
--image input.jpg --prompt "A cat playing with yarn" \
@@ -79,17 +87,20 @@ parse_profiler_config = functools.partial(parse_json_object, flag_name="--profil
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Generate a video from one or more images (Wan2.2, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE)."
description=(
"Generate a video from one or more images "
"(Wan2.2, LTX2/LTX-2.3, HunyuanVideo-1.5, Cosmos3, or Wan2.1 VACE)."
)
)
parser.add_argument(
"--model",
default="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
help="Diffusers I2V model ID or local path (Wan2.2, HunyuanVideo-1.5, or Wan2.1 VACE).",
help="Diffusers I2V model ID or local path (Wan2.2, LTX2/LTX-2.3, HunyuanVideo-1.5, Cosmos3, or Wan2.1 VACE).",
)
parser.add_argument(
"--model-class-name",
default=None,
help="Override model class name (e.g., LTX2ImageToVideoPipeline).",
help="Override model class name (e.g., LTX2ImageToVideoPipeline or LTX23ImageToVideoPipeline).",
)
parser.add_argument("--image", help="Path to the first-frame or source image.")
parser.add_argument("--last-image", help="Path to a last-frame condition (used by models such as VACE).")
@@ -113,7 +124,10 @@ def parse_args() -> argparse.Namespace:
parser.add_argument("--width", type=int, default=None, help="Video width (auto-calculated from image if not set).")
parser.add_argument("--num-frames", type=int, default=None, help="Number of frames. Default: model-specific.")
parser.add_argument(
"--num-inference-steps", type=int, default=None, help="Sampling steps. Default: model-specific."
"--num-inference-steps",
type=int,
default=None,
help="Sampling steps. Default: model-specific.",
)
parser.add_argument("--boundary-ratio", type=float, default=0.875, help="Boundary split ratio for MoE models.")
parser.add_argument(
@@ -312,10 +326,8 @@ def main():
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
model_name = str(args.model).lower() if args.model is not None else ""
model_class_name = args.model_class_name
is_ltx2 = "ltx2" in model_name or (model_class_name and "ltx2" in model_class_name.lower())
is_ltx2 = model_class_name in {"LTX2ImageToVideoPipeline", "LTX23ImageToVideoPipeline"}
is_cosmos = "cosmos" in model_name or (model_class_name is not None and "cosmos" in model_class_name.lower())
if model_class_name is None and is_ltx2:
model_class_name = "LTX2ImageToVideoPipeline"
image = PIL.Image.open(args.image).convert("RGB") if args.image else None
last_image = PIL.Image.open(args.last_image).convert("RGB") if args.last_image else None
@@ -340,7 +352,15 @@ def main():
16,
)
elif is_ltx2:
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = 24, 4.0, 121, 40, 5.0, 512 * 768, 32
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = (
24,
4.0,
121,
40,
5.0,
512 * 768,
32,
)
else: # Wan2.2 / HunyuanVideo-1.5
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = 16, 5.0, 81, 50, 5.0, 480 * 832, 16
@@ -462,7 +482,7 @@ def main():
f" tensor_parallel_size={args.tensor_parallel_size}, vae_patch_parallel_size={args.vae_patch_parallel_size},"
f" pipeline_parallel_size={args.pipeline_parallel_size}"
)
print(f" Video size: {args.width}x{args.height}")
print(f" Video size: {width}x{height}")
print(f"{'=' * 60}\n")
prompt_dict = build_image_to_video_prompt(
@@ -509,6 +529,7 @@ def main():
print(f"Total generation time: {generation_time:.4f} seconds ({generation_time * 1000:.2f} ms)")
audio = None
audio_sample_rate = args.audio_sample_rate
if isinstance(frames, list):
frames = frames[0] if frames else None
@@ -519,11 +540,13 @@ def main():
)
if frames.multimodal_output and "audio" in frames.multimodal_output:
audio = frames.multimodal_output["audio"]
audio_sample_rate = frames.multimodal_output.get("audio_sample_rate", audio_sample_rate)
if frames.is_pipeline_output and frames.request_output is not None:
inner_output = frames.request_output
if isinstance(inner_output, OmniRequestOutput):
if inner_output.multimodal_output and "audio" in inner_output.multimodal_output:
audio = inner_output.multimodal_output["audio"]
audio_sample_rate = inner_output.multimodal_output.get("audio_sample_rate", audio_sample_rate)
frames = inner_output
if isinstance(frames, OmniRequestOutput):
if frames.images:
@@ -531,6 +554,7 @@ def main():
frames, audio = frames.images[0]
elif len(frames.images) == 1 and isinstance(frames.images[0], dict):
audio = frames.images[0].get("audio")
audio_sample_rate = frames.images[0].get("audio_sample_rate", audio_sample_rate)
frames = frames.images[0].get("frames") or frames.images[0].get("video")
else:
frames = frames.images
@@ -543,6 +567,7 @@ def main():
frames, audio = first_item
elif isinstance(first_item, dict):
audio = first_item.get("audio")
audio_sample_rate = first_item.get("audio_sample_rate", audio_sample_rate)
frames = first_item.get("frames") or first_item.get("video")
elif isinstance(first_item, list):
frames = first_item
@@ -551,6 +576,7 @@ def main():
frames, audio = frames
elif isinstance(frames, dict):
audio = frames.get("audio")
audio_sample_rate = frames.get("audio_sample_rate", audio_sample_rate)
frames = frames.get("frames") or frames.get("video")
if frames is None:
@@ -669,7 +695,7 @@ def main():
frames_u8,
audio_np,
fps=float(fps),
audio_sample_rate=args.audio_sample_rate,
audio_sample_rate=audio_sample_rate,
)
with open(str(output_path), "wb") as f:
f.write(video_bytes)
examples/offline_inference/text_to_video/text_to_video.md
+29 -1
View File
@@ -8,6 +8,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
|---|---|---|---|---|---|
| `Wan-AI/Wan2.1-VACE-1.3B-diffusers` | 480x832 | 81 | 30 | 5.0 | ~20 GiB (RTX 5090, VAE tiling) |
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB |
| `dg845/LTX-2.3-Diffusers` | 384x512 | 25 | 20 | 4.0 | 96GB-class GPU |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
| `nvidia/Cosmos3-Nano` | 720x1280 | 189 | 35 | 6.0 | ~46 GiB (peak, 720p) |
@@ -70,6 +71,30 @@ python text_to_video.py \
--output ltx2_out.mp4
```
### LTX-2.3
```bash
python text_to_video.py \
--model dg845/LTX-2.3-Diffusers \
--model-class-name LTX23Pipeline \
--prompt "Cherry blossoms swaying gently in the breeze with synchronized ambient sound" \
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
--height 384 \
--width 512 \
--num-frames 25 \
--num-inference-steps 20 \
--guidance-scale 4.0 \
--frame-rate 24 \
--fps 24 \
--audio-sample-rate 48000 \
--output ltx23_t2v_output.mp4
```
Use a Diffusers-format checkpoint such as `dg845/LTX-2.3-Diffusers`; the
upstream `Lightricks/LTX-2.3` raw safetensors repo is not directly loadable by
this pipeline. Pass `--model-class-name LTX23Pipeline` to select the LTX-2.3
text-to-video pipeline explicitly.
### HunyuanVideo-1.5 (480p)
```bash
@@ -187,6 +212,8 @@ python text_to_video.py \
### Common
- `--model`: Diffusers model ID or local path.
- `--model-class-name`: Optional explicit pipeline class. Use `LTX23Pipeline`
for LTX-2.3 text-to-video.
- `--prompt`: text description (string).
- `--height/--width`: output resolution. Default depends on model.
- `--num-frames`: number of frames. Default depends on model.
@@ -202,7 +229,8 @@ python text_to_video.py \
- `--enable-cpu-offload`: enable CPU offloading for diffusion models.
- `--enable-layerwise-offload`: enable layerwise offloading on DiT modules.
- `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2).
- `--audio-sample-rate`: audio sample rate for embedded audio (when the pipeline returns audio).
- `--audio-sample-rate`: audio sample rate for embedded audio (when the
pipeline returns audio; LTX-2.3 outputs 48kHz audio).
- `--quantization`: quantization method (`fp8` for FP8, `gguf` for GGUF).
- `--flow-shift`: scheduler flow_shift parameter.
- `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3 (see above).
examples/offline_inference/text_to_video/text_to_video.py
+7 -1
View File
@@ -455,6 +455,7 @@ def main():
print(f"Worker peak GPU memory (reserved): {peak_mb:.2f} MiB ({peak_mb / 1024:.2f} GiB)")
audio = None
audio_sample_rate = args.audio_sample_rate
if isinstance(frames, list):
frames = frames[0] if frames else None
@@ -465,11 +466,13 @@ def main():
)
if frames.multimodal_output and "audio" in frames.multimodal_output:
audio = frames.multimodal_output["audio"]
audio_sample_rate = frames.multimodal_output.get("audio_sample_rate", audio_sample_rate)
if frames.is_pipeline_output and frames.request_output is not None:
inner_output = frames.request_output
if isinstance(inner_output, OmniRequestOutput):
if inner_output.multimodal_output and "audio" in inner_output.multimodal_output:
audio = inner_output.multimodal_output["audio"]
audio_sample_rate = inner_output.multimodal_output.get("audio_sample_rate", audio_sample_rate)
frames = inner_output
if isinstance(frames, OmniRequestOutput):
if frames.images:
@@ -477,6 +480,7 @@ def main():
frames, audio = frames.images[0]
elif len(frames.images) == 1 and isinstance(frames.images[0], dict):
audio = frames.images[0].get("audio")
audio_sample_rate = frames.images[0].get("audio_sample_rate", audio_sample_rate)
frames = frames.images[0].get("frames") or frames.images[0].get("video")
else:
frames = frames.images
@@ -489,6 +493,7 @@ def main():
frames, audio = first_item
elif isinstance(first_item, dict):
audio = first_item.get("audio")
audio_sample_rate = first_item.get("audio_sample_rate", audio_sample_rate)
frames = first_item.get("frames") or first_item.get("video")
elif isinstance(first_item, list):
frames = first_item
@@ -497,6 +502,7 @@ def main():
frames, audio = frames
elif isinstance(frames, dict):
audio = frames.get("audio")
audio_sample_rate = frames.get("audio_sample_rate", audio_sample_rate)
frames = frames.get("frames") or frames.get("video")
if frames is None:
@@ -611,7 +617,7 @@ def main():
frames_u8,
audio_np,
fps=float(args.fps),
audio_sample_rate=args.audio_sample_rate,
audio_sample_rate=audio_sample_rate,
)
with open(str(output_path), "wb") as f:
f.write(video_bytes)