[Feature] Add official LTX multimodal guidance and precision guard (#5148)

Signed-off-by: mglyn <1203789601@qq.com>
This commit is contained in:
Mu GuanLin authored and GitHub committed 2026-07-28 10:44:25 +00:00
1 parent 66a9b0c84f
commit 12d75cf63a
51 files changed
+4547 -3062

No files matched your search

@@ -248,82 +248,12 @@ done
## LTX-2
### Start Server
#### Basic Start
```bash
vllm serve Lightricks/LTX-2 --omni --port 8098 \
--enforce-eager --boundary-ratio 1.0
vllm serve Lightricks/LTX-2 --omni --port 8098
```
#### Start with Optimization Presets
Use the LTX-2 startup script with built-in optimization presets:
```bash
# Baseline (1 GPU, eager)
bash run_server_ltx2.sh baseline
# 4-GPU Ulysses sequence parallelism (lossless)
bash run_server_ltx2.sh ulysses4
# Cache-DiT lossy acceleration (1 GPU, ~1.4× speedup)
bash run_server_ltx2.sh cache-dit
# Best combo: 4-GPU Ulysses SP + Cache-DiT (~2.2× speedup)
bash run_server_ltx2.sh best-combo
```
#### Optimization Benchmarks
Benchmarked on H800, online serving (480×768, 41 frames, 20 steps, `seed=42`).
"Inference" is the server-reported inference time; excludes HTTP/poll overhead.
| Preset | Server Command | Inference (s) | Speedup | Type |
|--------|---------------|---------------|---------|------|
| `baseline` | `--enforce-eager` | 10.3 | 1.00× | — |
| `compile` | *(default, no --enforce-eager)* | ~10.3 (warm) | ~1.00× | Lossless |
| `ulysses4` | `--enforce-eager --usp 4` | ~10.3 | ~1.00× | Lossless |
| `cache-dit` | `--enforce-eager --cache-backend cache_dit` | 7.4 avg | ~1.4× | Lossy |
| `best-combo` | `--enforce-eager --usp 4 --cache-backend cache_dit` | 4.7 avg | **~2.2×** | Lossless + Lossy |
**Observations**:
- **torch.compile**: On H800, warm-request inference time matches the eager baseline (~10.3s).
The first request pays ~6s compilation overhead. Benefit depends on model architecture and GPU.
- **Ulysses SP (4 GPU)**: No measurable speedup alone for 41-frame generation at this resolution.
Communication overhead outweighs gains at this sequence length.
- **Cache-DiT**: Inference varies per request (6–10s) due to dynamic caching decisions.
Average is ~7.4s (~1.4× speedup) with slight quality tradeoff.
- **Best combo**: 4-GPU Ulysses SP + Cache-DiT synergize well — Cache-DiT reduces per-step
computation, making the communication overhead of Ulysses SP worthwhile. Average ~4.7s
(~2.2× speedup).
- **FP8 quantization**: Reduces VRAM but does not speed up LTX-2 on H800 (compute-bound).
**Deployment Recommendations**:
- For **production with quality priority**: use `baseline` with `--enforce-eager`
- For **maximum throughput** (4 GPUs, quality tradeoff): use `best-combo` (~2.2× speedup)
- For **single-GPU throughput**: use `cache-dit` (~1.4× speedup)
- `--enforce-eager` is recommended to avoid torch.compile warmup latency on first request
### Send Requests (curl)
```bash
# Using the provided script
bash run_curl_ltx2.sh
# Or directly
curl -sS -X POST http://localhost:8098/v1/videos \
-H "Accept: application/json" \
-F "prompt=A serene lakeside sunrise with mist over the water." \
-F "width=768" \
-F "height=480" \
-F "num_frames=41" \
-F "fps=24" \
-F "num_inference_steps=20" \
-F "guidance_scale=3.0" \
-F "seed=42"
```
See the [LTX-2 recipe](../../../recipes/LTX/LTX-2.md) for all checkpoints,
pipeline selection, requests, defaults, and advanced options.
## Helios
@@ -22,7 +22,6 @@ create_response=$(
-F "num_frames=41" \
-F "fps=24" \
-F "num_inference_steps=20" \
-F "guidance_scale=3.0" \
-F "seed=42"
)
@@ -21,8 +21,6 @@ set -euo pipefail
MODEL="${MODEL:-Lightricks/LTX-2}"
PORT="${PORT:-8098}"
FLOW_SHIFT="${FLOW_SHIFT:-1.0}"
BOUNDARY_RATIO="${BOUNDARY_RATIO:-1.0}"
PRESET="${1:-baseline}"
@@ -66,19 +64,13 @@ case "$PRESET" in
echo "Environment variables:"
echo " MODEL - Model path (default: Lightricks/LTX-2)"
echo " PORT - Server port (default: 8098)"
echo " FLOW_SHIFT - Scheduler flow shift (default: 1.0)"
echo " BOUNDARY_RATIO - Boundary ratio (default: 1.0)"
exit 1
;;
esac
echo "Model: $MODEL"
echo "Port: $PORT"
echo "Flow shift: $FLOW_SHIFT"
echo "Boundary ratio: $BOUNDARY_RATIO"
vllm serve "$MODEL" --omni \
--port "$PORT" \
--flow-shift "$FLOW_SHIFT" \
--boundary-ratio "$BOUNDARY_RATIO" \
"${EXTRA_ARGS[@]}"