diff --git a/apps/lvsa-omni-showcase/README.md b/apps/lvsa-omni-showcase/README.md new file mode 100644 index 000000000..69596fa3c --- /dev/null +++ b/apps/lvsa-omni-showcase/README.md @@ -0,0 +1,87 @@ +# LVSA Long-Video Showcase + +**Training-free block-sparse attention for long-video diffusion**, running on +vLLM-Omni via the public plugin entry points (no core changes). LVSA speeds up +extended-horizon generation and prevents the freeze/loop failure mode dense +attention exhibits beyond a model's training length — with no fine-tuning and no +weight changes. + +Upstream: **[LongVideoSparseAttention](https://github.com/JiusiServe/LongVideoSparseAttention)** +(algorithm, all model adapters, full docs). + +## Results + +Wan 2.1 1.3B, single A100 80 GB, mean over 5 prompts (Dense vs LVSA-FlashInfer): + +| Horizon | 1× | 2× | 3× | 4× | 5× | 6× | +|---|---|---|---|---|---|---| +| **Speedup** | 0.9× | 1.4× | 1.9× | 2.5× | 3.0× | **3.5×** | +| **VQeval composite** (Dense→LVSA) | 63→63 | 59→63 | 61→63 | 61→64 | 59→64 | 59→63 | + +The speedup **grows with length** (LVSA is the dense regime at 1× by design, so +the win starts at 2×), and quality *improves* at extension — LVSA's rotating +keyframes prevent the looping/static output dense produces beyond its horizon +(VQeval loop-quality +27 to +34 at ≥3×). At long horizons dense **OOMs on 80 GB** +(HunyuanVideo ≥2×, Wan2.1-14B and Cosmos at 6×) where LVSA still generates. + +→ Full sweep (5 models × 6 horizons, VQeval + VBench-Long) and the +quality-metric methodology: +**[benchmarks/](https://github.com/JiusiServe/LongVideoSparseAttention/tree/main/benchmarks)**. + +## Quickstart + +```bash +# 1. vLLM-Omni 0.22.0 (stable) + matching vLLM +pip install "vllm==0.22.0" +pip install --no-build-isolation \ + "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@v0.22.0" + +# 2. LVSA core + plugin +git clone https://github.com/JiusiServe/LongVideoSparseAttention.git +( cd LongVideoSparseAttention && pip install -e . && pip install -e lvsa-vllm-omni/ ) + +# 3. this showcase's client dep +pip install requests +``` + +FlashInfer (used by the default `flashinfer` backend) **ships with vLLM** — step 1 +already pulls the pinned `flashinfer-python` + `flashinfer-cubin`, so the fused +backend works with no extra install. Two optional extras: + +```bash +# (optional) faster startup — prebuilt JIT cache, matched to your CUDA + flashinfer +CU=cu128 # match your CUDA: cu121 / cu124 / cu128 / … +FIVER=$(python -c "import importlib.metadata as m; print(m.version('flashinfer-python'))") +pip install --extra-index-url "https://flashinfer.ai/whl/${CU}" \ + "flashinfer-jit-cache==${FIVER}+${CU}" + +# (fallback) no FlashInfer in your environment? run the SDPA backend instead: +export LVSA_BACKEND=sdpa +``` + +```bash +# Serve an LVSA-enabled endpoint (Wan 2.1 1.3B at 2× horizon): +MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers MODEL_FAMILY=wan FRAMES=161 \ + bash serve_lvsa.sh + +# In another shell — generate: +python generate.py --frames 161 --height 480 --width 832 \ + --prompt "A dog running through a sunlit forest." --out wan_lvsa.mp4 +``` + +`serve_lvsa.sh` sets the LVSA env vars for the chosen `MODEL_FAMILY` +(`wan` / `hunyuan`). Wan and HunyuanVideo run through the LVSA **attention +backend** by default — it stays sparse even under sequence-parallel, unlike the +monkey-patch hooks. + +## Configuration + +The one knob you must get right is the model's training horizon +(`LVSA_REFERENCE_LATENT_FRAMES`: Wan = 21, HunyuanVideo = 33); `serve_lvsa.sh` +sets it per family. For the full environment-variable reference, the multi-GPU +support matrix, and tuning (`sparsity_scale`, rotating keyframes, non-standard +resolutions), see the upstream docs: + +- **[lvsa-vllm-omni plugin reference](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/lvsa-vllm-omni/README.md)** — every `LVSA_*` env var +- **[docs/parallelism.md](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/docs/parallelism.md)** — TP / Ulysses / HSDP support per model +- **[docs/tuning.md](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/docs/tuning.md)** — picking knobs for your model and horizon diff --git a/apps/lvsa-omni-showcase/generate.py b/apps/lvsa-omni-showcase/generate.py new file mode 100644 index 000000000..32de4310a --- /dev/null +++ b/apps/lvsa-omni-showcase/generate.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python +"""Generate a long video through an LVSA-enabled vLLM-Omni server. + +A thin client over the Omni video API — start the server first with +``serve_lvsa.sh``. The sparse attention runs server-side; this just submits a +job and saves the result. + +Uses the **asynchronous** endpoint (``POST /v1/videos`` → poll +``GET /v1/videos/{id}`` → download ``GET /v1/videos/{id}/content``) rather than +``POST /v1/videos/sync``. The sync endpoint is capped by a fixed server-side +timeout (``VIDEO_SYNC_TIMEOUT_S``, 600 s) and is documented as +"for benchmark and testing scenarios" — long videos (LVSA's use case) routinely +exceed it. The async job has no such cap. + + python generate.py --frames 481 --height 480 --width 832 \ + --prompt "A golden retriever running through a sunlit forest." \ + --out dog_forest_lvsa.mp4 +""" + +import argparse +import os +import time + +import requests + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--server", default=os.environ.get("OMNI_SERVER", "http://localhost:8000")) + ap.add_argument("--prompt", default="A dog running through a sunlit forest.") + ap.add_argument("--frames", type=int, default=161) # Wan 2x horizon + ap.add_argument("--height", type=int, default=480) + ap.add_argument("--width", type=int, default=832) + ap.add_argument("--fps", type=int, default=16) + ap.add_argument("--steps", type=int, default=50) + ap.add_argument("--guidance", type=float, default=5.0) + ap.add_argument("--out", default="lvsa_long_video.mp4") + ap.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between status polls.") + ap.add_argument( + "--max-wait", type=float, default=7200.0, help="Give up if the job hasn't finished after this many seconds." + ) + args = ap.parse_args() + + base = args.server.rstrip("/") + form = { + "prompt": args.prompt, + "size": f"{args.width}x{args.height}", + "num_frames": str(args.frames), + "fps": str(args.fps), + "num_inference_steps": str(args.steps), + "guidance_scale": str(args.guidance), + } + + # ── 1. submit the async job ────────────────────────────────────────────── + print(f"[lvsa-app] POST {base}/v1/videos {args.width}x{args.height} {args.frames}f steps={args.steps}") + resp = requests.post(f"{base}/v1/videos", data=form, timeout=120) + resp.raise_for_status() + job = resp.json() + vid = job["id"] + print(f"[lvsa-app] job {vid} ({job.get('status', '?')})") + + # ── 2. poll until completed / failed ───────────────────────────────────── + deadline = time.time() + args.max_wait + last = None + while True: + if time.time() > deadline: + raise SystemExit(f"[lvsa-app] timed out after {args.max_wait:.0f}s (job {vid} still running)") + time.sleep(args.poll_interval) + r = requests.get(f"{base}/v1/videos/{vid}", timeout=60) + if r.status_code != 200: + # retrieve_video returns a non-200 JSON when the job has FAILED + raise SystemExit(f"[lvsa-app] job failed: {r.text}") + st = r.json() + status = st.get("status") + progress = st.get("progress", 0) + if (status, progress) != last: + print(f"[lvsa-app] {status} {progress}%") + last = (status, progress) + if status == "completed": + break + if status == "failed": + raise SystemExit(f"[lvsa-app] job failed: {st.get('error')}") + + # ── 3. download the rendered mp4 ───────────────────────────────────────── + c = requests.get(f"{base}/v1/videos/{vid}/content", timeout=600) + c.raise_for_status() + with open(args.out, "wb") as f: + f.write(c.content) + print(f"[lvsa-app] saved {args.out}") + + +if __name__ == "__main__": + main() diff --git a/apps/lvsa-omni-showcase/serve_lvsa.sh b/apps/lvsa-omni-showcase/serve_lvsa.sh new file mode 100644 index 000000000..58b71d14c --- /dev/null +++ b/apps/lvsa-omni-showcase/serve_lvsa.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Start an LVSA-enabled vLLM-Omni server for Wan or HunyuanVideo. +# +# LVSA is selected as the attention backend via --diffusion-attention-config (see +# below); the `LVSA_*` env vars only tune it. The `lvsa-vllm-omni` plugin registers +# itself through vLLM-Omni's `general_plugins` entry points — nothing in vLLM-Omni +# core is modified. Watch the log for "[LVSA] Geometry detected" to confirm sparse +# attention is active for your request geometry. +# +# MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers MODEL_FAMILY=wan FRAMES=161 bash serve_lvsa.sh +# MODEL=/path/to/HunyuanVideo-1.5-... MODEL_FAMILY=hunyuan FRAMES=193 bash serve_lvsa.sh +set -euo pipefail + +MODEL=${MODEL:?set MODEL=/path/to/Wan-or-HunyuanVideo-Diffusers} +MODEL_FAMILY=${MODEL_FAMILY:-wan} # wan | hunyuan +PORT=${PORT:-8000} +FRAMES=${FRAMES:-161} # target horizon in frames +VAE_T=${VAE_T:-4} # VAE temporal compression (Wan/Hunyuan = 4) +BACKEND=${LVSA_BACKEND:-flashinfer} # flashinfer (fused, fastest) | sdpa + +T_LAT=$(( (FRAMES - 1) / VAE_T + 1 )) + +# Per-family training reference horizon (Wan train ref = 21, Hunyuan = 33). +case "$MODEL_FAMILY" in + wan) REFERENCE=${REFERENCE:-21} ;; + hunyuan) REFERENCE=${REFERENCE:-33} ;; + *) echo "MODEL_FAMILY must be 'wan' or 'hunyuan'"; exit 1 ;; +esac +export LVSA_BACKEND="$BACKEND" +export LVSA_TOTAL_LATENT_FRAMES="$T_LAT" +export LVSA_REFERENCE_LATENT_FRAMES="$REFERENCE" +export LVSA_ROTATE_KEYFRAMES=${LVSA_ROTATE_KEYFRAMES:-1} + +echo "[lvsa-app] serving $MODEL ($MODEL_FAMILY) on :$PORT" +echo "[lvsa-app] LVSA backend=$BACKEND T_lat=$T_LAT (frames=$FRAMES) reference=$REFERENCE" +echo "[lvsa-app] confirm activation by grep '\\[LVSA\\] Geometry detected' in this log" + +# vLLM-Omni 0.22 selects the LVSA attention backend per-role via +# --diffusion-attention-config. The backend is what drives sparse attention +# here; it engages under tensor-parallel too, unlike the LVSA_*_HOOK monkey-patch +# path (which falls back to dense under sequence-parallel), so we deliberately do +# NOT set a hook. +exec vllm serve "$MODEL" --omni --host 0.0.0.0 --port "$PORT" \ + --diffusion-attention-config '{"per_role": {"self": {"backend": "LVSA"}}}' \ + --init-timeout 1800 ${EXTRA_ARGS:-}