From d89dc0ad5869b945cea799c56f51f1bc6cdf3bce Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 19 Jan 2026 05:48:28 +0000 Subject: [PATCH] Fix ReadTheDocs build: move LTX-2 docs to proper location and add to nav Co-authored-by: hsliuustc0106 <222337142+hsliuustc0106@users.noreply.github.com> --- docs/.nav.yml | 1 + docs/LTX2_INTEGRATION.md | 173 ------------------ docs/models/supported_models.md | 1 + .../examples/offline_inference/ltx2.md | 42 +++++ 4 files changed, 44 insertions(+), 173 deletions(-) delete mode 100644 docs/LTX2_INTEGRATION.md create mode 100644 docs/user_guide/examples/offline_inference/ltx2.md diff --git a/docs/.nav.yml b/docs/.nav.yml index be9306371..2d8bf87ef 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -12,6 +12,7 @@ nav: - Offline Inference: - Image-To-Image: user_guide/examples/offline_inference/image_to_image.md - Image-To-Video: user_guide/examples/offline_inference/image_to_video.md + - LTX-2: user_guide/examples/offline_inference/ltx2.md - Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md - Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md - Text-To-Image: user_guide/examples/offline_inference/text_to_image.md diff --git a/docs/LTX2_INTEGRATION.md b/docs/LTX2_INTEGRATION.md deleted file mode 100644 index 22d7dbb41..000000000 --- a/docs/LTX2_INTEGRATION.md +++ /dev/null @@ -1,173 +0,0 @@ -# LTX-2 Model Integration - -## Overview - -This integration adds support for the Lightricks LTX-2 text-to-video generation model to vllm-omni. LTX-2 is a state-of-the-art open-source model capable of generating high-quality videos (up to 4K resolution) from text prompts or images. - -## Model Information - -- **Model ID**: `Lightricks/LTX-2` -- **Model Type**: Text-to-Video (T2V) and Image-to-Video (I2V) -- **Architecture**: Diffusion Transformer (DiT) with 19B parameters -- **HuggingFace**: https://huggingface.co/Lightricks/LTX-2 -- **GitHub**: https://github.com/Lightricks/LTX-2 - -## Quick Start - -### Basic Usage - -```python -from vllm_omni.entrypoints.omni import Omni - -# Initialize the model -omni = Omni(model="Lightricks/LTX-2") - -# Generate a video -output = omni.generate( - prompt="A panda riding a bicycle through a forest, cinematic lighting", - height=512, - width=768, - num_frames=121, - num_inference_steps=40, - guidance_scale=4.0, -) -``` - -### Using the Example Script - -```bash -cd examples/offline_inference/ltx2 - -python text_to_video.py \ - --prompt "A serene lakeside sunrise with mist over the water" \ - --height 512 \ - --width 768 \ - --num_frames 121 \ - --output my_video.mp4 -``` - -See the [example documentation](examples/offline_inference/ltx2/text_to_video.md) for more details. - -## Parameters - -### Video Dimensions -- **height**: Video height in pixels (must be divisible by 32) -- **width**: Video width in pixels (must be divisible by 32) -- Default: 512x768 - -### Frame Count -- **num_frames**: Number of frames (should follow pattern 8*n+1) -- Valid examples: 25, 41, 81, 121, 161 -- Default: 121 frames (approximately 5 seconds at 24fps) - -### Generation Quality -- **num_inference_steps**: Number of denoising steps (default: 40) - - More steps = higher quality but slower generation - - Range: 20-50 steps typically - -- **guidance_scale**: Classifier-free guidance strength (default: 4.0) - - Higher values follow prompt more strictly - - Range: 3.0-5.0 recommended - -### Other Options -- **negative_prompt**: Text describing what to avoid in the video -- **seed**: Random seed for reproducibility -- **fps**: Output video frame rate (default: 24) - -## Implementation Details - -### Architecture - -The integration follows vllm-omni's diffusion model patterns: - -``` -vllm_omni/diffusion/models/ltx2/ -├── __init__.py # Module exports -└── pipeline_ltx2.py # Main pipeline implementation -``` - -### Key Components - -1. **LTX2Pipeline**: Wraps diffusers' `LTXVideoPipeline` - - Handles parameter validation - - Ensures dimension constraints (divisible by 32) - - Manages frame count convention (8*n+1) - -2. **Pre-processing**: `get_ltx2_pre_process_func` - - Loads and validates image inputs for I2V mode - -3. **Post-processing**: `get_ltx2_post_process_func` - - Converts video tensors to numpy arrays - - Handles different output formats - -### Registry Integration - -The model is registered in `vllm_omni/diffusion/registry.py`: -- Model class: `LTX2Pipeline` -- Pre-process function: `get_ltx2_pre_process_func` -- Post-process function: `get_ltx2_post_process_func` - -## Requirements - -- **Python**: >= 3.10 -- **GPU**: CUDA-capable GPU with >= 24GB VRAM recommended -- **Dependencies**: diffusers >= 0.36.0 (already included in vllm-omni) - -### Model Download - -On first use, the model will be automatically downloaded from HuggingFace: -- Size: Approximately 20-30GB -- Location: `~/.cache/huggingface/hub/` - -## Performance Tips - -1. **Start Small**: Test with lower resolutions (512x768) and fewer frames (25-81) -2. **Increase Gradually**: Scale up resolution and frame count based on your GPU -3. **Optimize Steps**: 40 steps provide good quality; adjust based on speed/quality needs -4. **Batch Processing**: For multiple videos, reuse the same Omni instance to avoid reloading - -## Troubleshooting - -### Out of Memory (OOM) - -If you encounter OOM errors: -- Reduce video dimensions (e.g., 480x640 instead of 512x768) -- Reduce frame count (e.g., 25 or 41 instead of 121) -- Enable VAE tiling/slicing if supported - -### Slow Generation - -For faster generation: -- Reduce `num_inference_steps` (e.g., 30 instead of 40) -- Use smaller dimensions -- Generate fewer frames - -### Quality Issues - -For better quality: -- Increase `num_inference_steps` (e.g., 50) -- Adjust `guidance_scale` (try 3.5-5.0) -- Use more descriptive prompts with style keywords - -## Example Prompts - -```python -# Cinematic nature scene -"A serene lakeside sunrise with mist over the water, cinematic, high quality" - -# Action scene -"A futuristic cityscape with flying cars, neon lights, cyberpunk style, 4k" - -# Animated style -"A cartoon panda eating bamboo in a forest, Pixar style animation" - -# Close-up shot -"Close-up of raindrops falling on a window, soft focus, moody lighting" -``` - -## References - -- [LTX-2 on HuggingFace](https://huggingface.co/Lightricks/LTX-2) -- [LTX-2 GitHub Repository](https://github.com/Lightricks/LTX-2) -- [Diffusers Documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video) -- [Example Usage](examples/offline_inference/ltx2/text_to_video.md) diff --git a/docs/models/supported_models.md b/docs/models/supported_models.md index cd70019de..cd5c4e99e 100644 --- a/docs/models/supported_models.md +++ b/docs/models/supported_models.md @@ -34,6 +34,7 @@ th { |`StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | |`Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | |`StableAudioPipeline` | Stable-Audio-Open | `stabilityai/stable-audio-open-1.0` | +|`LTX2Pipeline` | LTX-2 | `Lightricks/LTX-2` | ## List of Supported Models for NPU diff --git a/docs/user_guide/examples/offline_inference/ltx2.md b/docs/user_guide/examples/offline_inference/ltx2.md new file mode 100644 index 000000000..ffa6f312b --- /dev/null +++ b/docs/user_guide/examples/offline_inference/ltx2.md @@ -0,0 +1,42 @@ +# LTX-2 Text-To-Video + +Source . + +The `Lightricks/LTX-2` pipeline generates high-quality videos from text prompts using a 19B parameter DiT architecture. It supports up to 4K video generation with synchronized audio. + +## Local CLI Usage + +```bash +python text_to_video.py \ + --prompt "A panda riding a bicycle through a forest, cinematic lighting" \ + --height 512 \ + --width 768 \ + --num_frames 121 \ + --num_inference_steps 40 \ + --guidance_scale 4.0 \ + --output ltx2_output.mp4 +``` + +Key arguments: + +- `--prompt`: Text description of the video to generate (string). +- `--height/--width`: Output resolution (defaults 512x768). Dimensions must be divisible by 32. +- `--num_frames`: Number of frames (should be 8*n+1, e.g., 25, 81, 121). Default is 121. +- `--guidance_scale`: Classifier-free guidance scale (default: 4.0). Range 3.0-5.0 recommended. +- `--negative_prompt`: Optional text describing what to avoid in the video. +- `--num_inference_steps`: Number of denoising steps (default: 40). +- `--seed`: Random seed for reproducibility (default: 42). +- `--fps`: Frames per second for the saved MP4 (default: 24). +- `--output`: Path to save the generated video. + +## Example materials + +??? abstract "text_to_video.py" + ``````py + --8<-- "examples/offline_inference/ltx2/text_to_video.py" + `````` + +??? abstract "text_to_video.md" + ``````md + --8<-- "examples/offline_inference/ltx2/text_to_video.md" + ``````