anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Fix ReadTheDocs build: move LTX-2 docs to proper location and add to nav

Browse Source 
Co-authored-by: hsliuustc0106 <222337142+hsliuustc0106@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]andhsliuustc0106 committed 2026-01-19 05:48:28 +00:00
1 parent 31187ff519
commit d89dc0ad58
4 files changed
+44 -173
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/.nav.yml
+1
View File
@@ -12,6 +12,7 @@ nav:
- Offline Inference:
- Image-To-Image: user_guide/examples/offline_inference/image_to_image.md
- Image-To-Video: user_guide/examples/offline_inference/image_to_video.md
- LTX-2: user_guide/examples/offline_inference/ltx2.md
- Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md
- Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md
- Text-To-Image: user_guide/examples/offline_inference/text_to_image.md
docs/LTX2_INTEGRATION.md
-173
View File
@@ -1,173 +0,0 @@
# LTX-2 Model Integration
## Overview
This integration adds support for the Lightricks LTX-2 text-to-video generation model to vllm-omni. LTX-2 is a state-of-the-art open-source model capable of generating high-quality videos (up to 4K resolution) from text prompts or images.
## Model Information
- **Model ID**: `Lightricks/LTX-2`
- **Model Type**: Text-to-Video (T2V) and Image-to-Video (I2V)
- **Architecture**: Diffusion Transformer (DiT) with 19B parameters
- **HuggingFace**: https://huggingface.co/Lightricks/LTX-2
- **GitHub**: https://github.com/Lightricks/LTX-2
## Quick Start
### Basic Usage
```python
from vllm_omni.entrypoints.omni import Omni
# Initialize the model
omni = Omni(model="Lightricks/LTX-2")
# Generate a video
output = omni.generate(
prompt="A panda riding a bicycle through a forest, cinematic lighting",
height=512,
width=768,
num_frames=121,
num_inference_steps=40,
guidance_scale=4.0,
)
```
### Using the Example Script
```bash
cd examples/offline_inference/ltx2
python text_to_video.py \
--prompt "A serene lakeside sunrise with mist over the water" \
--height 512 \
--width 768 \
--num_frames 121 \
--output my_video.mp4
```
See the [example documentation](examples/offline_inference/ltx2/text_to_video.md) for more details.
## Parameters
### Video Dimensions
- **height**: Video height in pixels (must be divisible by 32)
- **width**: Video width in pixels (must be divisible by 32)
- Default: 512x768
### Frame Count
- **num_frames**: Number of frames (should follow pattern 8*n+1)
- Valid examples: 25, 41, 81, 121, 161
- Default: 121 frames (approximately 5 seconds at 24fps)
### Generation Quality
- **num_inference_steps**: Number of denoising steps (default: 40)
- More steps = higher quality but slower generation
- Range: 20-50 steps typically
- **guidance_scale**: Classifier-free guidance strength (default: 4.0)
- Higher values follow prompt more strictly
- Range: 3.0-5.0 recommended
### Other Options
- **negative_prompt**: Text describing what to avoid in the video
- **seed**: Random seed for reproducibility
- **fps**: Output video frame rate (default: 24)
## Implementation Details
### Architecture
The integration follows vllm-omni's diffusion model patterns:
```
vllm_omni/diffusion/models/ltx2/
├── __init__.py # Module exports
└── pipeline_ltx2.py # Main pipeline implementation
```
### Key Components
1. **LTX2Pipeline**: Wraps diffusers' `LTXVideoPipeline`
- Handles parameter validation
- Ensures dimension constraints (divisible by 32)
- Manages frame count convention (8*n+1)
2. **Pre-processing**: `get_ltx2_pre_process_func`
- Loads and validates image inputs for I2V mode
3. **Post-processing**: `get_ltx2_post_process_func`
- Converts video tensors to numpy arrays
- Handles different output formats
### Registry Integration
The model is registered in `vllm_omni/diffusion/registry.py`:
- Model class: `LTX2Pipeline`
- Pre-process function: `get_ltx2_pre_process_func`
- Post-process function: `get_ltx2_post_process_func`
## Requirements
- **Python**: >= 3.10
- **GPU**: CUDA-capable GPU with >= 24GB VRAM recommended
- **Dependencies**: diffusers >= 0.36.0 (already included in vllm-omni)
### Model Download
On first use, the model will be automatically downloaded from HuggingFace:
- Size: Approximately 20-30GB
- Location: `~/.cache/huggingface/hub/`
## Performance Tips
1. **Start Small**: Test with lower resolutions (512x768) and fewer frames (25-81)
2. **Increase Gradually**: Scale up resolution and frame count based on your GPU
3. **Optimize Steps**: 40 steps provide good quality; adjust based on speed/quality needs
4. **Batch Processing**: For multiple videos, reuse the same Omni instance to avoid reloading
## Troubleshooting
### Out of Memory (OOM)
If you encounter OOM errors:
- Reduce video dimensions (e.g., 480x640 instead of 512x768)
- Reduce frame count (e.g., 25 or 41 instead of 121)
- Enable VAE tiling/slicing if supported
### Slow Generation
For faster generation:
- Reduce `num_inference_steps` (e.g., 30 instead of 40)
- Use smaller dimensions
- Generate fewer frames
### Quality Issues
For better quality:
- Increase `num_inference_steps` (e.g., 50)
- Adjust `guidance_scale` (try 3.5-5.0)
- Use more descriptive prompts with style keywords
## Example Prompts
```python
# Cinematic nature scene
"A serene lakeside sunrise with mist over the water, cinematic, high quality"
# Action scene
"A futuristic cityscape with flying cars, neon lights, cyberpunk style, 4k"
# Animated style
"A cartoon panda eating bamboo in a forest, Pixar style animation"
# Close-up shot
"Close-up of raindrops falling on a window, soft focus, moody lighting"
```
## References
- [LTX-2 on HuggingFace](https://huggingface.co/Lightricks/LTX-2)
- [LTX-2 GitHub Repository](https://github.com/Lightricks/LTX-2)
- [Diffusers Documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video)
- [Example Usage](examples/offline_inference/ltx2/text_to_video.md)
docs/models/supported_models.md
+1
View File
@@ -34,6 +34,7 @@ th {
|`StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` |
|`Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` |
|`StableAudioPipeline` | Stable-Audio-Open | `stabilityai/stable-audio-open-1.0` |
|`LTX2Pipeline` | LTX-2 | `Lightricks/LTX-2` |
## List of Supported Models for NPU
docs/user_guide/examples/offline_inference/ltx2.md
+42
View File
@@ -0,0 +1,42 @@
# LTX-2 Text-To-Video
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/ltx2>.
The `Lightricks/LTX-2` pipeline generates high-quality videos from text prompts using a 19B parameter DiT architecture. It supports up to 4K video generation with synchronized audio.
## Local CLI Usage
```bash
python text_to_video.py \
--prompt "A panda riding a bicycle through a forest, cinematic lighting" \
--height 512 \
--width 768 \
--num_frames 121 \
--num_inference_steps 40 \
--guidance_scale 4.0 \
--output ltx2_output.mp4
```
Key arguments:
- `--prompt`: Text description of the video to generate (string).
- `--height/--width`: Output resolution (defaults 512x768). Dimensions must be divisible by 32.
- `--num_frames`: Number of frames (should be 8*n+1, e.g., 25, 81, 121). Default is 121.
- `--guidance_scale`: Classifier-free guidance scale (default: 4.0). Range 3.0-5.0 recommended.
- `--negative_prompt`: Optional text describing what to avoid in the video.
- `--num_inference_steps`: Number of denoising steps (default: 40).
- `--seed`: Random seed for reproducibility (default: 42).
- `--fps`: Frames per second for the saved MP4 (default: 24).
- `--output`: Path to save the generated video.
## Example materials
??? abstract "text_to_video.py"
``````py
--8<-- "examples/offline_inference/ltx2/text_to_video.py"
``````
??? abstract "text_to_video.md"
``````md
--8<-- "examples/offline_inference/ltx2/text_to_video.md"
``````