diff --git a/docs/LTX2_INTEGRATION.md b/docs/LTX2_INTEGRATION.md new file mode 100644 index 000000000..c157666fc --- /dev/null +++ b/docs/LTX2_INTEGRATION.md @@ -0,0 +1,173 @@ +# LTX-2 Model Integration + +## Overview + +This integration adds support for the Lightricks LTX-2 text-to-video generation model to vllm-omni. LTX-2 is a state-of-the-art open-source model capable of generating high-quality videos (up to 4K resolution) from text prompts or images. + +## Model Information + +- **Model ID**: `Lightricks/LTX-2` +- **Model Type**: Text-to-Video (T2V) and Image-to-Video (I2V) +- **Architecture**: Diffusion Transformer (DiT) with 19B parameters +- **HuggingFace**: https://huggingface.co/Lightricks/LTX-2 +- **GitHub**: https://github.com/Lightricks/LTX-2 + +## Quick Start + +### Basic Usage + +```python +from vllm_omni.entrypoints.omni import Omni + +# Initialize the model +omni = Omni(model="Lightricks/LTX-2") + +# Generate a video +output = omni.generate( + prompt="A panda riding a bicycle through a forest, cinematic lighting", + height=512, + width=768, + num_frames=121, + num_inference_steps=40, + guidance_scale=4.0, +) +``` + +### Using the Example Script + +```bash +cd examples/offline_inference/ltx2 + +python text_to_video.py \ + --prompt "A serene lakeside sunrise with mist over the water" \ + --height 512 \ + --width 768 \ + --num_frames 121 \ + --output my_video.mp4 +``` + +See the [example documentation](examples/offline_inference/ltx2/text_to_video.md) for more details. + +## Parameters + +### Video Dimensions +- **height**: Video height in pixels (must be divisible by 32) +- **width**: Video width in pixels (must be divisible by 32) +- Default: 512x768 + +### Frame Count +- **num_frames**: Number of frames (should follow pattern 8*n+1) +- Valid examples: 25, 41, 81, 121, 161 +- Default: 121 frames (approximately 5 seconds at 24fps) + +### Generation Quality +- **num_inference_steps**: Number of denoising steps (default: 40) + - More steps = higher quality but slower generation + - Range: 20-50 steps typically + +- **guidance_scale**: Classifier-free guidance strength (default: 4.0) + - Higher values follow prompt more strictly + - Range: 3.0-5.0 recommended + +### Other Options +- **negative_prompt**: Text describing what to avoid in the video +- **seed**: Random seed for reproducibility +- **fps**: Output video frame rate (default: 24) + +## Implementation Details + +### Architecture + +The integration follows vllm-omni's diffusion model patterns: + +``` +vllm_omni/diffusion/models/ltx2/ +├── __init__.py # Module exports +└── pipeline_ltx2.py # Main pipeline implementation +``` + +### Key Components + +1. **LTX2Pipeline**: Wraps diffusers' `LTXVideoPipeline` + - Handles parameter validation + - Ensures dimension constraints (divisible by 32) + - Manages frame count convention (8*n+1) + +2. **Pre-processing**: `get_ltx2_pre_process_func` + - Loads and validates image inputs for I2V mode + +3. **Post-processing**: `get_ltx2_post_process_func` + - Converts video tensors to numpy arrays + - Handles different output formats + +### Registry Integration + +The model is registered in `vllm_omni/diffusion/registry.py`: +- Model class: `LTX2Pipeline` +- Pre-process function: `get_ltx2_pre_process_func` +- Post-process function: `get_ltx2_post_process_func` + +## Requirements + +- **Python**: >= 3.10 +- **GPU**: CUDA-capable GPU with >= 24GB VRAM recommended +- **Dependencies**: diffusers >= 0.36.0 (already included in vllm-omni) + +### Model Download + +On first use, the model will be automatically downloaded from HuggingFace: +- Size: Approximately 20-30GB +- Location: `~/.cache/huggingface/hub/` + +## Performance Tips + +1. **Start Small**: Test with lower resolutions (512x768) and fewer frames (25-81) +2. **Increase Gradually**: Scale up resolution and frame count based on your GPU +3. **Optimize Steps**: 40 steps provide good quality; adjust based on speed/quality needs +4. **Batch Processing**: For multiple videos, reuse the same Omni instance to avoid reloading + +## Troubleshooting + +### Out of Memory (OOM) + +If you encounter OOM errors: +- Reduce video dimensions (e.g., 480x640 instead of 512x768) +- Reduce frame count (e.g., 25 or 41 instead of 121) +- Enable VAE tiling/slicing if supported + +### Slow Generation + +For faster generation: +- Reduce `num_inference_steps` (e.g., 30 instead of 40) +- Use smaller dimensions +- Generate fewer frames + +### Quality Issues + +For better quality: +- Increase `num_inference_steps` (e.g., 50) +- Adjust `guidance_scale` (try 3.5-5.0) +- Use more descriptive prompts with style keywords + +## Example Prompts + +```python +# Cinematic nature scene +"A serene lakeside sunrise with mist over the water, cinematic, high quality" + +# Action scene +"A futuristic cityscape with flying cars, neon lights, cyberpunk style, 4k" + +# Animated style +"A cartoon panda eating bamboo in a forest, Pixar style animation" + +# Close-up shot +"Close-up of raindrops falling on a window, soft focus, moody lighting" +``` + +## References + +- [LTX-2 on HuggingFace](https://huggingface.co/Lightricks/LTX-2) +- [LTX-2 GitHub Repository](https://github.com/Lightricks/LTX-2) +- [Diffusers Documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video) +- [Example Usage](examples/offline_inference/ltx2/text_to_video.md)