vllm-omni/docs/LTX2_INTEGRATION.md

LTX-2 Model Integration

Overview

This integration adds support for the Lightricks LTX-2 text-to-video generation model to vllm-omni. LTX-2 is a state-of-the-art open-source model capable of generating high-quality videos (up to 4K resolution) from text prompts or images.

Model Information

• Model ID: Lightricks/LTX-2
• Model Type: Text-to-Video (T2V) and Image-to-Video (I2V)
• Architecture: Diffusion Transformer (DiT) with 19B parameters
• HuggingFace: https://huggingface.co/Lightricks/LTX-2
• GitHub: https://github.com/Lightricks/LTX-2

Quick Start

Basic Usage

from vllm_omni.entrypoints.omni import Omni

# Initialize the model
omni = Omni(model="Lightricks/LTX-2")

# Generate a video
output = omni.generate(
    prompt="A panda riding a bicycle through a forest, cinematic lighting",
    height=512,
    width=768,
    num_frames=121,
    num_inference_steps=40,
    guidance_scale=4.0,
)

Using the Example Script

cd examples/offline_inference/ltx2

python text_to_video.py \
    --prompt "A serene lakeside sunrise with mist over the water" \
    --height 512 \
    --width 768 \
    --num_frames 121 \
    --output my_video.mp4

See the example documentation for more details.

Parameters

Video Dimensions

• height: Video height in pixels (must be divisible by 32)
• width: Video width in pixels (must be divisible by 32)
• Default: 512x768

Frame Count

• num_frames: Number of frames (should follow pattern 8*n+1)
• Valid examples: 25, 41, 81, 121, 161
• Default: 121 frames (approximately 5 seconds at 24fps)

Generation Quality

• num_inference_steps: Number of denoising steps (default: 40)

  • More steps = higher quality but slower generation
  • Range: 20-50 steps typically

• guidance_scale: Classifier-free guidance strength (default: 4.0)

  • Higher values follow prompt more strictly
  • Range: 3.0-5.0 recommended

Other Options

• negative_prompt: Text describing what to avoid in the video
• seed: Random seed for reproducibility
• fps: Output video frame rate (default: 24)

Implementation Details

Architecture

The integration follows vllm-omni's diffusion model patterns:

vllm_omni/diffusion/models/ltx2/
├── __init__.py                  # Module exports
└── pipeline_ltx2.py             # Main pipeline implementation

Key Components

1. LTX2Pipeline: Wraps diffusers' LTXVideoPipeline

   • Handles parameter validation
   • Ensures dimension constraints (divisible by 32)
   • Manages frame count convention (8*n+1)

2. Pre-processing: get_ltx2_pre_process_func

   • Loads and validates image inputs for I2V mode

3. Post-processing: get_ltx2_post_process_func

   • Converts video tensors to numpy arrays
   • Handles different output formats

Registry Integration

The model is registered in vllm_omni/diffusion/registry.py:

• Model class: LTX2Pipeline
• Pre-process function: get_ltx2_pre_process_func
• Post-process function: get_ltx2_post_process_func

Requirements

• Python: >= 3.10
• GPU: CUDA-capable GPU with >= 24GB VRAM recommended
• Dependencies: diffusers >= 0.36.0 (already included in vllm-omni)

Model Download

On first use, the model will be automatically downloaded from HuggingFace:

• Size: Approximately 20-30GB
• Location: ~/.cache/huggingface/hub/

Performance Tips

1. Start Small: Test with lower resolutions (512x768) and fewer frames (25-81)
2. Increase Gradually: Scale up resolution and frame count based on your GPU
3. Optimize Steps: 40 steps provide good quality; adjust based on speed/quality needs
4. Batch Processing: For multiple videos, reuse the same Omni instance to avoid reloading

Troubleshooting

Out of Memory (OOM)

If you encounter OOM errors:

• Reduce video dimensions (e.g., 480x640 instead of 512x768)
• Reduce frame count (e.g., 25 or 41 instead of 121)
• Enable VAE tiling/slicing if supported

Slow Generation

For faster generation:

• Reduce num_inference_steps (e.g., 30 instead of 40)
• Use smaller dimensions
• Generate fewer frames

Quality Issues

For better quality:

• Increase num_inference_steps (e.g., 50)
• Adjust guidance_scale (try 3.5-5.0)
• Use more descriptive prompts with style keywords

Example Prompts

# Cinematic nature scene
"A serene lakeside sunrise with mist over the water, cinematic, high quality"

# Action scene
"A futuristic cityscape with flying cars, neon lights, cyberpunk style, 4k"

# Animated style
"A cartoon panda eating bamboo in a forest, Pixar style animation"

# Close-up shot
"Close-up of raindrops falling on a window, soft focus, moody lighting"

References

• LTX-2 on HuggingFace
• LTX-2 GitHub Repository
• Diffusers Documentation
• Example Usage