[Feature] Add official LTX multimodal guidance and precision guard (#5148)
Signed-off-by: mglyn <1203789601@qq.com>
This commit is contained in:
1 parent
66a9b0c84f
commit
12d75cf63a
51 files changed
+4547
-3062
No files matched your search
@@ -1,7 +1,7 @@
|
||||
# Image-To-Video
|
||||
|
||||
This shared example generates videos from images with VACE, Wan2.2,
|
||||
LTX-2/LTX-2.3, HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
|
||||
This shared example generates videos from images with VACE, Wan2.2, LTX-2,
|
||||
HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
|
||||
|
||||
- `image_to_video.py`: command-line script for single video generation with advanced options.
|
||||
|
||||
@@ -26,7 +26,7 @@ This folder provides a unified CLI script for image-to-video generation using vL
|
||||
| `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480 x 832 | 81 | 50 | 5.0 | Around 60 GiB BF16 for basic single-card usage |
|
||||
| `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 480 x 832 | 81 | 50 | 4.0 | Around 20–25 GiB BF16, smallest I2V model |
|
||||
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v` | 480 x 832 | 121 | 50 | 6.0 | Around 100 GiB at default settings; the example enables `--enable-cpu-offload` + VAE tiling/slicing to fit an 80 GiB card |
|
||||
| LTX2 (local path + `--model-class-name LTX2ImageToVideoPipeline`) | 512 x 768 | 121 | 40 | 4.0 | Memory use depends on frame count and tensor parallelism |
|
||||
| `Lightricks/LTX-2` | 512 x 768 | 121 | 40 | video 3.0 / audio 7.0 | Memory use depends on frame count and tensor parallelism |
|
||||
|
||||
!!! info
|
||||
Peak VRAM: based on basic single-card usage, batch size = 1, without any acceleration/optimization features. Some model weights cannot fit into one card with 80 GiB VRAM, which may need to use CPU offloading.
|
||||
@@ -105,7 +105,7 @@ python image_to_video.py \
|
||||
| Argument | Type | Default | Description |
|
||||
| -------- | ---- | ------- | ----------- |
|
||||
| `--model` | str | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Diffusers I2V model ID or local path |
|
||||
| `--model-class-name` | str | `None` | Override model class name (e.g., `LTX2ImageToVideoPipeline`) |
|
||||
| `--model-class-name` | str | `None` | Optional pipeline override |
|
||||
| `--image` | str | (required) | Path to input image |
|
||||
| `--prompt` | str | `""` | Text description of desired motion/animation |
|
||||
| `--negative-prompt` | str | `""` | Optional list of artifacts to suppress |
|
||||
@@ -195,24 +195,19 @@ python image_to_video.py \
|
||||
--output hunyuan_i2v.mp4
|
||||
```
|
||||
|
||||
### LTX2 Image-to-Video
|
||||
### LTX-2
|
||||
|
||||
```bash
|
||||
python image_to_video.py \
|
||||
--model /path/to/LTX-2 \
|
||||
--model-class-name LTX2ImageToVideoPipeline \
|
||||
--model Lightricks/LTX-2 \
|
||||
--image cherry_blossom.jpg \
|
||||
--prompt "A cinematic dolly shot of cherry blossoms" \
|
||||
--height 512 \
|
||||
--width 768 \
|
||||
--num-frames 121 \
|
||||
--num-inference-steps 40 \
|
||||
--guidance-scale 4.0 \
|
||||
--frame-rate 24 \
|
||||
--fps 24 \
|
||||
--output ltx2_i2v.mp4
|
||||
```
|
||||
|
||||
See the [LTX-2 recipe](../../../recipes/LTX/LTX-2.md) for all checkpoints,
|
||||
pipeline selection, T2V, defaults, and advanced options.
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### CFG Parallel
|
||||
@@ -233,30 +228,6 @@ python image_to_video.py \
|
||||
--output i2v_cached.mp4
|
||||
```
|
||||
|
||||
### LTX-2.3
|
||||
|
||||
```bash
|
||||
python image_to_video.py \
|
||||
--model diffusers/LTX-2.3-Diffusers \
|
||||
--model-class-name LTX23ImageToVideoPipeline \
|
||||
--image cherry_blossom.jpg \
|
||||
--prompt "Cherry blossoms swaying gently in the breeze with synchronized ambient sound" \
|
||||
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
|
||||
--height 384 \
|
||||
--width 512 \
|
||||
--num-frames 25 \
|
||||
--guidance-scale 4.0 \
|
||||
--num-inference-steps 20 \
|
||||
--frame-rate 24 \
|
||||
--fps 24 \
|
||||
--output ltx23_i2v_output.mp4
|
||||
```
|
||||
|
||||
Use the Diffusers-format checkpoint `diffusers/LTX-2.3-Diffusers`; the
|
||||
upstream `Lightricks/LTX-2.3` raw safetensors repo is not directly loadable by
|
||||
this pipeline. Pass `--model-class-name LTX23ImageToVideoPipeline` to select
|
||||
the LTX-2.3 image-to-video pipeline.
|
||||
|
||||
### Cosmos3
|
||||
|
||||
```bash
|
||||
@@ -274,8 +245,8 @@ python image_to_video.py \
|
||||
|
||||
Key arguments:
|
||||
|
||||
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V,
|
||||
LTX-2/LTX-2.3, Cosmos3, or VACE).
|
||||
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V, LTX-2,
|
||||
Cosmos3, or VACE).
|
||||
- `--image`: Path to the first-frame or source image.
|
||||
- `--last-image`: Optional last-frame condition for models such as VACE.
|
||||
- `--mask-image`: Optional inpainting mask. White pixels are regenerated and black pixels are preserved.
|
||||
@@ -290,7 +261,7 @@ Key arguments:
|
||||
- `--guidance-scale` and `--guidance-scale-high`: CFG scale (applied to low/high-noise stages for MoE).
|
||||
- `--negative-prompt`: Optional list of artifacts to suppress.
|
||||
- `--boundary-ratio`: Boundary split ratio for two-stage MoE models.
|
||||
- `--flow-shift`: Scheduler flow shift (default: model-specific — Wan/LTX2 5.0, Cosmos3 10.0).
|
||||
- `--flow-shift`: Scheduler flow shift (default: Wan 5.0, Cosmos3 10.0). LTX does not consume this option.
|
||||
- `--sample-solver`: Wan2.2 sampling solver. Use `unipc` for the default multistep solver, or `euler` for Lightning/Distill checkpoints.
|
||||
- `--num-inference-steps`: Number of denoising steps (default: model-specific — Wan 50, LTX2 40, Cosmos3 35).
|
||||
- `--fps`: Frames per second for the saved MP4 (requires `diffusers` export_to_video).
|
||||
|
||||
@@ -23,16 +23,14 @@ Usage:
|
||||
|
||||
# LTX2 image-to-video
|
||||
python image_to_video.py --model /path/to/LTX-2 \
|
||||
--model-class-name LTX2ImageToVideoPipeline \
|
||||
--image input.jpg --prompt "A cinematic dolly shot of a boat" \
|
||||
--num-frames 121 --num-inference-steps 40 --guidance-scale 4.0 \
|
||||
--num-frames 121 --num-inference-steps 40 \
|
||||
--frame-rate 24 --fps 24 --output ltx2_i2v.mp4
|
||||
|
||||
# LTX-2.3 image-to-video
|
||||
python image_to_video.py --model diffusers/LTX-2.3-Diffusers \
|
||||
--model-class-name LTX23ImageToVideoPipeline \
|
||||
--image input.jpg --prompt "A cinematic dolly shot of a boat" \
|
||||
--height 384 --width 512 --num-frames 25 --num-inference-steps 20 \
|
||||
--height 512 --width 768 --num-frames 121 --num-inference-steps 30 \
|
||||
--frame-rate 24 --fps 24 --output ltx23_i2v.mp4
|
||||
|
||||
# HunyuanVideo-1.5 I2V (480p)
|
||||
@@ -100,7 +98,7 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument(
|
||||
"--model-class-name",
|
||||
default=None,
|
||||
help="Override model class name (e.g., LTX2ImageToVideoPipeline or LTX23ImageToVideoPipeline).",
|
||||
help="Override model class name (LTX checkpoints default to LTX2Pipeline).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--deploy-config",
|
||||
@@ -117,7 +115,7 @@ def parse_args() -> argparse.Namespace:
|
||||
help="Path to a reference image. Repeat to provide multiple references.",
|
||||
)
|
||||
parser.add_argument("--prompt", default="", help="Text prompt describing the desired motion.")
|
||||
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
|
||||
parser.add_argument("--negative-prompt", default=None, help="Negative prompt. Default: model-specific.")
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
|
||||
parser.add_argument("--guidance-scale", type=float, default=None, help="CFG scale. Default: model-specific.")
|
||||
parser.add_argument(
|
||||
@@ -331,7 +329,10 @@ def main():
|
||||
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
|
||||
model_name = str(args.model).lower() if args.model is not None else ""
|
||||
model_class_name = args.model_class_name
|
||||
is_ltx2 = model_class_name in {"LTX2ImageToVideoPipeline", "LTX23ImageToVideoPipeline"}
|
||||
model_class_name_lower = (model_class_name or "").lower()
|
||||
is_ltx2_distilled = "distilled" in model_class_name_lower or "distilled" in model_name
|
||||
is_ltx23 = "ltx23" in model_class_name_lower or "ltx-2.3" in model_name
|
||||
is_ltx2 = is_ltx2_distilled or is_ltx23 or "ltx2" in model_class_name_lower or "ltx-2" in model_name
|
||||
is_cosmos = "cosmos" in model_name or (model_class_name is not None and "cosmos" in model_class_name.lower())
|
||||
|
||||
image = PIL.Image.open(args.image).convert("RGB") if args.image else None
|
||||
@@ -356,13 +357,23 @@ def main():
|
||||
1280 * 720,
|
||||
16,
|
||||
)
|
||||
elif is_ltx2_distilled:
|
||||
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = (
|
||||
24,
|
||||
None,
|
||||
121,
|
||||
8,
|
||||
None,
|
||||
1024 * 1536,
|
||||
64,
|
||||
)
|
||||
elif is_ltx2:
|
||||
d_fps, d_guidance, d_num_frames, d_steps, d_flow_shift, d_max_area, d_mod = (
|
||||
24,
|
||||
4.0,
|
||||
None,
|
||||
121,
|
||||
40,
|
||||
5.0,
|
||||
30 if is_ltx23 else 40,
|
||||
None,
|
||||
512 * 768,
|
||||
32,
|
||||
)
|
||||
@@ -445,7 +456,6 @@ def main():
|
||||
vae_use_slicing=args.vae_use_slicing,
|
||||
vae_use_tiling=args.vae_use_tiling,
|
||||
boundary_ratio=args.boundary_ratio,
|
||||
flow_shift=flow_shift,
|
||||
diffusion_kv_cache_dtype=args.diffusion_kv_cache_dtype,
|
||||
diffusion_kv_cache_skip_steps=args.diffusion_kv_cache_skip_steps,
|
||||
diffusion_kv_cache_skip_layers=args.diffusion_kv_cache_skip_layers,
|
||||
@@ -460,6 +470,8 @@ def main():
|
||||
)
|
||||
if args.deploy_config:
|
||||
omni_kwargs["deploy_config"] = args.deploy_config
|
||||
if flow_shift is not None:
|
||||
omni_kwargs["flow_shift"] = flow_shift
|
||||
if args.quantization is not None:
|
||||
omni_kwargs["quantization"] = args.quantization
|
||||
# Cosmos3 loads its (gated) guardrail models at build time, so the guardrails
|
||||
@@ -492,10 +504,14 @@ def main():
|
||||
print(f" Video size: {width}x{height}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
negative_prompt = args.negative_prompt
|
||||
if negative_prompt is None and not is_ltx2:
|
||||
# Preserve the historical empty-prompt behavior for non-LTX examples.
|
||||
negative_prompt = ""
|
||||
prompt_dict = build_image_to_video_prompt(
|
||||
model_class_name=model_class_name,
|
||||
prompt=args.prompt,
|
||||
negative_prompt=args.negative_prompt,
|
||||
negative_prompt=negative_prompt,
|
||||
media_inputs=media_inputs,
|
||||
height=height,
|
||||
width=width,
|
||||
@@ -511,11 +527,10 @@ def main():
|
||||
num_inference_steps=num_inference_steps,
|
||||
num_frames=num_frames,
|
||||
frame_rate=frame_rate,
|
||||
extra_args={
|
||||
"sample_solver": args.sample_solver,
|
||||
"flow_shift": flow_shift,
|
||||
},
|
||||
extra_args={"sample_solver": args.sample_solver},
|
||||
)
|
||||
if flow_shift is not None:
|
||||
sampling_params.extra_args["flow_shift"] = flow_shift
|
||||
|
||||
# Route model-specific knobs through extra_body, filtered against the model's
|
||||
# declared extra_body_params. Models without a declaration only forward explicit
|
||||
|
||||
@@ -8,7 +8,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
|
||||
|---|---|---|---|---|---|
|
||||
| `Wan-AI/Wan2.1-VACE-1.3B-diffusers` | 480x832 | 81 | 30 | 5.0 | ~20 GiB (RTX 5090, VAE tiling) |
|
||||
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB |
|
||||
| `diffusers/LTX-2.3-Diffusers` | 384x512 | 25 | 20 | 4.0 | 96GB-class GPU |
|
||||
| `Lightricks/LTX-2` | 512x768 | 121 | 40 | video 3.0 / audio 7.0 | Model-dependent |
|
||||
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
|
||||
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
|
||||
| `nvidia/Cosmos3-Nano` | 720x1280 | 189 | 35 | 6.0 | ~46 GiB (peak, 720p) |
|
||||
@@ -55,45 +55,17 @@ python text_to_video.py \
|
||||
--output vace_t2v_output.mp4
|
||||
```
|
||||
|
||||
LTX2 example:
|
||||
### LTX-2
|
||||
|
||||
```bash
|
||||
python text_to_video.py \
|
||||
--model "Lightricks/LTX-2" \
|
||||
--prompt "A cinematic close-up of ocean waves at golden hour." \
|
||||
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
|
||||
--height 512 \
|
||||
--width 768 \
|
||||
--num-frames 121 \
|
||||
--num-inference-steps 40 \
|
||||
--guidance-scale 4.0 \
|
||||
--frame-rate 24 \
|
||||
--output ltx2_out.mp4
|
||||
```
|
||||
|
||||
### LTX-2.3
|
||||
|
||||
```bash
|
||||
python text_to_video.py \
|
||||
--model diffusers/LTX-2.3-Diffusers \
|
||||
--model-class-name LTX23Pipeline \
|
||||
--model Lightricks/LTX-2 \
|
||||
--prompt "Cherry blossoms swaying gently in the breeze with synchronized ambient sound" \
|
||||
--negative-prompt "worst quality, inconsistent motion, blurry, jittery, distorted" \
|
||||
--height 384 \
|
||||
--width 512 \
|
||||
--num-frames 25 \
|
||||
--num-inference-steps 20 \
|
||||
--guidance-scale 4.0 \
|
||||
--frame-rate 24 \
|
||||
--fps 24 \
|
||||
--audio-sample-rate 48000 \
|
||||
--output ltx23_t2v_output.mp4
|
||||
--output ltx2_output.mp4
|
||||
```
|
||||
|
||||
Use the Diffusers-format checkpoint `diffusers/LTX-2.3-Diffusers`; the
|
||||
upstream `Lightricks/LTX-2.3` raw safetensors repo is not directly loadable by
|
||||
this pipeline. Pass `--model-class-name LTX23Pipeline` to select the LTX-2.3
|
||||
text-to-video pipeline explicitly.
|
||||
See the [LTX-2 recipe](../../../recipes/LTX/LTX-2.md) for all checkpoints,
|
||||
pipeline selection, I2V, defaults, and advanced options.
|
||||
|
||||
### HunyuanVideo-1.5 (480p)
|
||||
|
||||
@@ -212,8 +184,7 @@ python text_to_video.py \
|
||||
### Common
|
||||
|
||||
- `--model`: Diffusers model ID or local path.
|
||||
- `--model-class-name`: Optional explicit pipeline class. Use `LTX23Pipeline`
|
||||
for LTX-2.3 text-to-video.
|
||||
- `--model-class-name`: Optional explicit pipeline override.
|
||||
- `--prompt`: text description (string).
|
||||
- `--height/--width`: output resolution. Default depends on model.
|
||||
- `--num-frames`: number of frames. Default depends on model.
|
||||
@@ -229,8 +200,7 @@ python text_to_video.py \
|
||||
- `--enable-cpu-offload`: enable CPU offloading for diffusion models.
|
||||
- `--enable-layerwise-offload`: enable layerwise offloading on DiT modules.
|
||||
- `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2).
|
||||
- `--audio-sample-rate`: audio sample rate for embedded audio (when the
|
||||
pipeline returns audio; LTX-2.3 outputs 48kHz audio).
|
||||
- `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio.
|
||||
- `--quantization`: quantization method (such as `fp8` for FP8).
|
||||
- `--flow-shift`: scheduler flow_shift parameter.
|
||||
- `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3 (see above).
|
||||
|
||||
@@ -65,11 +65,42 @@ _MODEL_PRESETS = {
|
||||
"fps": 16,
|
||||
"output": "helios_output.mp4",
|
||||
},
|
||||
"ltx2": {
|
||||
"height": 512,
|
||||
"width": 768,
|
||||
"num_frames": 121,
|
||||
"num_inference_steps": 40,
|
||||
"fps": 24,
|
||||
"output": "ltx2_output.mp4",
|
||||
},
|
||||
"ltx2_distilled": {
|
||||
"height": 1024,
|
||||
"width": 1536,
|
||||
"num_frames": 121,
|
||||
"num_inference_steps": 8,
|
||||
"fps": 24,
|
||||
"output": "ltx2_distilled_output.mp4",
|
||||
},
|
||||
"ltx23": {
|
||||
"height": 512,
|
||||
"width": 768,
|
||||
"num_frames": 121,
|
||||
"num_inference_steps": 30,
|
||||
"fps": 24,
|
||||
"output": "ltx23_output.mp4",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _detect_preset(model: str) -> dict:
|
||||
def _detect_preset(model: str, model_class_name: str | None = None) -> dict:
|
||||
model_lower = model.lower()
|
||||
class_lower = (model_class_name or "").lower()
|
||||
if "distilled" in class_lower or "distilled" in model_lower:
|
||||
return _MODEL_PRESETS["ltx2_distilled"]
|
||||
if "ltx23" in class_lower or "ltx-2.3" in model_lower or "ltx_2.3" in model_lower:
|
||||
return _MODEL_PRESETS["ltx23"]
|
||||
if "ltx2" in class_lower or "ltx-2" in model_lower or "ltx_2" in model_lower:
|
||||
return _MODEL_PRESETS["ltx2"]
|
||||
if "vace" in model_lower:
|
||||
return _MODEL_PRESETS["vace"]
|
||||
if "cosmos" in model_lower:
|
||||
@@ -122,7 +153,7 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument(
|
||||
"--model-class-name",
|
||||
default=None,
|
||||
help="Override model class name (e.g., LTX2TwoStagesVideoPipeline).",
|
||||
help="Override model class name (e.g., LTX2Pipeline).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--deploy-config",
|
||||
@@ -130,7 +161,7 @@ def parse_args() -> argparse.Namespace:
|
||||
help="Optional deploy config YAML to use for pipeline-backed runs.",
|
||||
)
|
||||
parser.add_argument("--prompt", default="A serene lakeside sunrise with mist over the water.", help="Text prompt.")
|
||||
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
|
||||
parser.add_argument("--negative-prompt", default=None, help="Negative prompt. Default: model-specific.")
|
||||
parser.add_argument(
|
||||
"--extra-body",
|
||||
type=parse_extra_body,
|
||||
@@ -331,7 +362,7 @@ def main():
|
||||
args = parse_args()
|
||||
model_class_name = args.model_class_name
|
||||
|
||||
preset = _detect_preset(args.model)
|
||||
preset = _detect_preset(args.model, model_class_name)
|
||||
for key, default_val in preset.items():
|
||||
if getattr(args, key.replace("-", "_"), None) is None:
|
||||
setattr(args, key.replace("-", "_"), default_val)
|
||||
@@ -422,8 +453,11 @@ def main():
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
prompt_dict = {"prompt": args.prompt}
|
||||
if args.negative_prompt:
|
||||
if args.negative_prompt is not None:
|
||||
prompt_dict["negative_prompt"] = args.negative_prompt
|
||||
elif preset not in (_MODEL_PRESETS["ltx2"], _MODEL_PRESETS["ltx23"]):
|
||||
# Preserve the historical empty-prompt behavior for non-LTX examples.
|
||||
prompt_dict["negative_prompt"] = ""
|
||||
|
||||
sampling_kwargs = dict(
|
||||
height=args.height,
|
||||
|
||||
Reference in new issue
Block a user