[Feature] Add official LTX multimodal guidance and precision guard (#5148)

Signed-off-by: mglyn <1203789601@qq.com>
This commit is contained in:
Mu GuanLin authored and GitHub committed 2026-07-28 10:44:25 +00:00
1 parent 66a9b0c84f
commit 12d75cf63a
51 files changed
+4547 -3062

No files matched your search

@@ -1,7 +1,7 @@
# Image-To-Video
This shared example generates videos from images with VACE, Wan2.2,
LTX-2/LTX-2.3, HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
This shared example generates videos from images with VACE, Wan2.2, LTX-2,
HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
- `image_to_video.py`: command-line script for single video generation with advanced options.
@@ -26,7 +26,7 @@ This folder provides a unified CLI script for image-to-video generation using vL
| `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480 x 832 | 81 | 50 | 5.0 | Around 60 GiB BF16 for basic single-card usage |
| `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 480 x 832 | 81 | 50 | 4.0 | Around 20–25 GiB BF16, smallest I2V model |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v` | 480 x 832 | 121 | 50 | 6.0 | Around 100 GiB at default settings; the example enables `--enable-cpu-offload` + VAE tiling/slicing to fit an 80 GiB card |
| LTX2 (local path + `--model-class-name LTX2ImageToVideoPipeline`) | 512 x 768 | 121 | 40 | 4.0 | Memory use depends on frame count and tensor parallelism |
| `Lightricks/LTX-2` | 512 x 768 | 121 | 40 | video 3.0 / audio 7.0 | Memory use depends on frame count and tensor parallelism |
!!! info
Peak VRAM: based on basic single-card usage, batch size = 1, without any acceleration/optimization features. Some model weights cannot fit into one card with 80 GiB VRAM, which may need to use CPU offloading.
@@ -105,7 +105,7 @@ python image_to_video.py \
| Argument | Type | Default | Description |
| -------- | ---- | ------- | ----------- |
| `--model` | str | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Diffusers I2V model ID or local path |
| `--model-class-name` | str | `None` | Override model class name (e.g., `LTX2ImageToVideoPipeline`) |
| `--model-class-name` | str | `None` | Optional pipeline override |
| `--image` | str | (required) | Path to input image |
| `--prompt` | str | `""` | Text description of desired motion/animation |
| `--negative-prompt` | str | `""` | Optional list of artifacts to suppress |
@@ -195,24 +195,19 @@ python image_to_video.py \
--output hunyuan_i2v.mp4
```
### LTX2 Image-to-Video
### LTX-2
```bash
python image_to_video.py \
--model /path/to/LTX-2 \
--model-class-name LTX2ImageToVideoPipeline \
--model Lightricks/LTX-2 \
--image cherry_blossom.jpg \
--prompt "A cinematic dolly shot of cherry blossoms" \
--height 512 \
--width 768 \
--num-frames 121 \
--num-inference-steps 40 \
--guidance-scale 4.0 \
--frame-rate 24 \
--fps 24 \
--output ltx2_i2v.mp4
```
See the [LTX-2 recipe](../../../recipes/LTX/LTX-2.md) for all checkpoints,
pipeline selection, T2V, defaults, and advanced options.
## Advanced Features
### CFG Parallel
@@ -233,30 +228,6 @@ python image_to_video.py \
--output i2v_cached.mp4
```
### LTX-2.3
```bash
python image_to_video.py \
--model diffusers/LTX-2.3-Diffusers \
--model-class-name LTX23ImageToVideoPipeline \
--image cherry_blossom.jpg \
--prompt "Cherry blossoms swaying gently in the breeze with synchronized ambient sound" \
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
--height 384 \
--width 512 \
--num-frames 25 \
--guidance-scale 4.0 \
--num-inference-steps 20 \
--frame-rate 24 \
--fps 24 \
--output ltx23_i2v_output.mp4
```
Use the Diffusers-format checkpoint `diffusers/LTX-2.3-Diffusers`; the
upstream `Lightricks/LTX-2.3` raw safetensors repo is not directly loadable by
this pipeline. Pass `--model-class-name LTX23ImageToVideoPipeline` to select
the LTX-2.3 image-to-video pipeline.
### Cosmos3
```bash
@@ -274,8 +245,8 @@ python image_to_video.py \
Key arguments:
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V,
LTX-2/LTX-2.3, Cosmos3, or VACE).
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V, LTX-2,
Cosmos3, or VACE).
- `--image`: Path to the first-frame or source image.
- `--last-image`: Optional last-frame condition for models such as VACE.
- `--mask-image`: Optional inpainting mask. White pixels are regenerated and black pixels are preserved.
@@ -290,7 +261,7 @@ Key arguments:
- `--guidance-scale` and `--guidance-scale-high`: CFG scale (applied to low/high-noise stages for MoE).
- `--negative-prompt`: Optional list of artifacts to suppress.
- `--boundary-ratio`: Boundary split ratio for two-stage MoE models.
- `--flow-shift`: Scheduler flow shift (default: model-specific — Wan/LTX2 5.0, Cosmos3 10.0).
- `--flow-shift`: Scheduler flow shift (default: Wan 5.0, Cosmos3 10.0). LTX does not consume this option.
- `--sample-solver`: Wan2.2 sampling solver. Use `unipc` for the default multistep solver, or `euler` for Lightning/Distill checkpoints.
- `--num-inference-steps`: Number of denoising steps (default: model-specific — Wan 50, LTX2 40, Cosmos3 35).
- `--fps`: Frames per second for the saved MP4 (requires `diffusers` export_to_video).
@@ -23,16 +23,14 @@ Usage:
# LTX2 image-to-video
python image_to_video.py --model /path/to/LTX-2 \
--model-class-name LTX2ImageToVideoPipeline \
--image input.jpg --prompt "A cinematic dolly shot of a boat" \
--num-frames 121 --num-inference-steps 40 --guidance-scale 4.0 \
--num-frames 121 --num-inference-steps 40 \
--frame-rate 24 --fps 24 --output ltx2_i2v.mp4
# LTX-2.3 image-to-video
python image_to_video.py --model diffusers/LTX-2.3-Diffusers \
--model-class-name LTX23ImageToVideoPipeline \
--image input.jpg --prompt "A cinematic dolly shot of a boat" \
--height 384 --width 512 --num-frames 25 --num-inference-steps 20 \
--height 512 --width 768 --num-frames 121 --num-inference-steps 30 \
--frame-rate 24 --fps 24 --output ltx23_i2v.mp4
# HunyuanVideo-1.5 I2V (480p)
@@ -100,7 +98,7 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--model-class-name",
default=None,
help="Override model class name (e.g., LTX2ImageToVideoPipeline or LTX23ImageToVideoPipeline).",
help="Override model class name (LTX checkpoints default to LTX2Pipeline).",
)
parser.add_argument(
"--deploy-config",
@@ -117,7 +115,7 @@ def parse_args() -> argparse.Namespace:
help="Path to a reference image. Repeat to provide multiple references.",
)
parser.add_argument("--prompt", default="", help="Text prompt describing the desired motion.")
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
parser.add_argument("--negative-prompt", default=None, help="Negative prompt. Default: model-specific.")
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
parser.add_argument("--guidance-scale", type=float, default=None, help="CFG scale. Default: model-specific.")
parser.add_argument(
@@ -331,7 +329,10 @@ def main():
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
model_name = str(args.model).lower() if args.model is not None else ""
model_class_name = args.model_class_name
is_ltx2 = model_class_name in {"LTX2ImageToVideoPipeline", "LTX23ImageToVideoPipeline"}
model_class_name_lower = (model_class_name or "").lower()
is_ltx2_distilled = "distilled" in model_class_name_lower or "distilled" in model_name
is_ltx23 = "ltx23" in model_class_name_lower or "ltx-2.3" in model_name
is_ltx2 = is_ltx2_distilled or is_ltx23 or "ltx2" in model_class_name_lower or "ltx-2" in model_name
is_cosmos = "cosmos" in model_name or (model_class_name is not None and "cosmos" in model_class_name.lower())
image = PIL.Image.open(args.image).convert("RGB") if args.image else None
@@ -356,13 +357,23 @@ def main():
1280 * 720,
16,
)
elif is_ltx2_distilled:
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = (
24,
None,
121,
8,
None,
1024 * 1536,
64,
)
elif is_ltx2:
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = (
24,
4.0,
None,
121,
40,
5.0,
30 if is_ltx23 else 40,
None,
512 * 768,
32,
)
@@ -445,7 +456,6 @@ def main():
vae_use_slicing=args.vae_use_slicing,
vae_use_tiling=args.vae_use_tiling,
boundary_ratio=args.boundary_ratio,
flow_shift=flow_shift,
diffusion_kv_cache_dtype=args.diffusion_kv_cache_dtype,
diffusion_kv_cache_skip_steps=args.diffusion_kv_cache_skip_steps,
diffusion_kv_cache_skip_layers=args.diffusion_kv_cache_skip_layers,
@@ -460,6 +470,8 @@ def main():
)
if args.deploy_config:
omni_kwargs["deploy_config"] = args.deploy_config
if flow_shift is not None:
omni_kwargs["flow_shift"] = flow_shift
if args.quantization is not None:
omni_kwargs["quantization"] = args.quantization
# Cosmos3 loads its (gated) guardrail models at build time, so the guardrails
@@ -492,10 +504,14 @@ def main():
print(f" Video size: {width}x{height}")
print(f"{'=' * 60}\n")
negative_prompt = args.negative_prompt
if negative_prompt is None and not is_ltx2:
# Preserve the historical empty-prompt behavior for non-LTX examples.
negative_prompt = ""
prompt_dict = build_image_to_video_prompt(
model_class_name=model_class_name,
prompt=args.prompt,
negative_prompt=args.negative_prompt,
negative_prompt=negative_prompt,
media_inputs=media_inputs,
height=height,
width=width,
@@ -511,11 +527,10 @@ def main():
num_inference_steps=num_inference_steps,
num_frames=num_frames,
frame_rate=frame_rate,
extra_args={
"sample_solver": args.sample_solver,
"flow_shift": flow_shift,
},
extra_args={"sample_solver": args.sample_solver},
)
if flow_shift is not None:
sampling_params.extra_args["flow_shift"] = flow_shift
# Route model-specific knobs through extra_body, filtered against the model's
# declared extra_body_params. Models without a declaration only forward explicit
@@ -8,7 +8,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
|---|---|---|---|---|---|
| `Wan-AI/Wan2.1-VACE-1.3B-diffusers` | 480x832 | 81 | 30 | 5.0 | ~20 GiB (RTX 5090, VAE tiling) |
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB |
| `diffusers/LTX-2.3-Diffusers` | 384x512 | 25 | 20 | 4.0 | 96GB-class GPU |
| `Lightricks/LTX-2` | 512x768 | 121 | 40 | video 3.0 / audio 7.0 | Model-dependent |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
| `nvidia/Cosmos3-Nano` | 720x1280 | 189 | 35 | 6.0 | ~46 GiB (peak, 720p) |
@@ -55,45 +55,17 @@ python text_to_video.py \
--output vace_t2v_output.mp4
```
LTX2 example:
### LTX-2
```bash
python text_to_video.py \
--model "Lightricks/LTX-2" \
--prompt "A cinematic close-up of ocean waves at golden hour." \
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
--height 512 \
--width 768 \
--num-frames 121 \
--num-inference-steps 40 \
--guidance-scale 4.0 \
--frame-rate 24 \
--output ltx2_out.mp4
```
### LTX-2.3
```bash
python text_to_video.py \
--model diffusers/LTX-2.3-Diffusers \
--model-class-name LTX23Pipeline \
--model Lightricks/LTX-2 \
--prompt "Cherry blossoms swaying gently in the breeze with synchronized ambient sound" \
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
--height 384 \
--width 512 \
--num-frames 25 \
--num-inference-steps 20 \
--guidance-scale 4.0 \
--frame-rate 24 \
--fps 24 \
--audio-sample-rate 48000 \
--output ltx23_t2v_output.mp4
--output ltx2_output.mp4
```
Use the Diffusers-format checkpoint `diffusers/LTX-2.3-Diffusers`; the
upstream `Lightricks/LTX-2.3` raw safetensors repo is not directly loadable by
this pipeline. Pass `--model-class-name LTX23Pipeline` to select the LTX-2.3
text-to-video pipeline explicitly.
See the [LTX-2 recipe](../../../recipes/LTX/LTX-2.md) for all checkpoints,
pipeline selection, I2V, defaults, and advanced options.
### HunyuanVideo-1.5 (480p)
@@ -212,8 +184,7 @@ python text_to_video.py \
### Common
- `--model`: Diffusers model ID or local path.
- `--model-class-name`: Optional explicit pipeline class. Use `LTX23Pipeline`
for LTX-2.3 text-to-video.
- `--model-class-name`: Optional explicit pipeline override.
- `--prompt`: text description (string).
- `--height/--width`: output resolution. Default depends on model.
- `--num-frames`: number of frames. Default depends on model.
@@ -229,8 +200,7 @@ python text_to_video.py \
- `--enable-cpu-offload`: enable CPU offloading for diffusion models.
- `--enable-layerwise-offload`: enable layerwise offloading on DiT modules.
- `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2).
- `--audio-sample-rate`: audio sample rate for embedded audio (when the
pipeline returns audio; LTX-2.3 outputs 48kHz audio).
- `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio.
- `--quantization`: quantization method (such as `fp8` for FP8).
- `--flow-shift`: scheduler flow_shift parameter.
- `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3 (see above).
@@ -65,11 +65,42 @@ _MODEL_PRESETS = {
"fps": 16,
"output": "helios_output.mp4",
},
"ltx2": {
"height": 512,
"width": 768,
"num_frames": 121,
"num_inference_steps": 40,
"fps": 24,
"output": "ltx2_output.mp4",
},
"ltx2_distilled": {
"height": 1024,
"width": 1536,
"num_frames": 121,
"num_inference_steps": 8,
"fps": 24,
"output": "ltx2_distilled_output.mp4",
},
"ltx23": {
"height": 512,
"width": 768,
"num_frames": 121,
"num_inference_steps": 30,
"fps": 24,
"output": "ltx23_output.mp4",
},
}
def _detect_preset(model: str) -> dict:
def _detect_preset(model: str, model_class_name: str | None = None) -> dict:
model_lower = model.lower()
class_lower = (model_class_name or "").lower()
if "distilled" in class_lower or "distilled" in model_lower:
return _MODEL_PRESETS["ltx2_distilled"]
if "ltx23" in class_lower or "ltx-2.3" in model_lower or "ltx_2.3" in model_lower:
return _MODEL_PRESETS["ltx23"]
if "ltx2" in class_lower or "ltx-2" in model_lower or "ltx_2" in model_lower:
return _MODEL_PRESETS["ltx2"]
if "vace" in model_lower:
return _MODEL_PRESETS["vace"]
if "cosmos" in model_lower:
@@ -122,7 +153,7 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--model-class-name",
default=None,
help="Override model class name (e.g., LTX2TwoStagesVideoPipeline).",
help="Override model class name (e.g., LTX2Pipeline).",
)
parser.add_argument(
"--deploy-config",
@@ -130,7 +161,7 @@ def parse_args() -> argparse.Namespace:
help="Optional deploy config YAML to use for pipeline-backed runs.",
)
parser.add_argument("--prompt", default="A serene lakeside sunrise with mist over the water.", help="Text prompt.")
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
parser.add_argument("--negative-prompt", default=None, help="Negative prompt. Default: model-specific.")
parser.add_argument(
"--extra-body",
type=parse_extra_body,
@@ -331,7 +362,7 @@ def main():
args = parse_args()
model_class_name = args.model_class_name
preset = _detect_preset(args.model)
preset = _detect_preset(args.model, model_class_name)
for key, default_val in preset.items():
if getattr(args, key.replace("-", "_"), None) is None:
setattr(args, key.replace("-", "_"), default_val)
@@ -422,8 +453,11 @@ def main():
print(f"{'=' * 60}\n")
prompt_dict = {"prompt": args.prompt}
if args.negative_prompt:
if args.negative_prompt is not None:
prompt_dict["negative_prompt"] = args.negative_prompt
elif preset not in (_MODEL_PRESETS["ltx2"], _MODEL_PRESETS["ltx23"]):
# Preserve the historical empty-prompt behavior for non-LTX examples.
prompt_dict["negative_prompt"] = ""
sampling_kwargs = dict(
height=args.height,