[skip ci][Misc] LVSA showcase (training-free block-sparse attention) (#4192)

Signed-off-by: Gael Glorian <gael.glorian@gmail.com> Signed-off-by: Ioannis Lamprou <yiannis.lamprou@gmail.com> Signed-off-by: Zhen Zhang <zhen.zhang.fr@huawei.com> Co-authored-by: Ioannis Lamprou <yiannis.lamprou@gmail.com> Co-authored-by: Zhen Zhang <zhen.zhang.fr@huawei.com>
2026-06-25 22:20:14 +08:00
1 parent 0b04f2a0d6
commit 36a5e9cfd2
@@ -0,0 +1,87 @@
+# LVSA Long-Video Showcase
+
+**Training-free block-sparse attention for long-video diffusion**, running on
+vLLM-Omni via the public plugin entry points (no core changes). LVSA speeds up
+extended-horizon generation and prevents the freeze/loop failure mode dense
+attention exhibits beyond a model's training length — with no fine-tuning and no
+weight changes.
+
+Upstream: **[LongVideoSparseAttention](https://github.com/JiusiServe/LongVideoSparseAttention)**
+(algorithm, all model adapters, full docs).
+
+## Results
+
+Wan 2.1 1.3B, single A100 80 GB, mean over 5 prompts (Dense vs LVSA-FlashInfer):
+
+| Horizon | 1× | 2× | 3× | 4× | 5× | 6× |
+|---|---|---|---|---|---|---|
+| **Speedup** | 0.9× | 1.4× | 1.9× | 2.5× | 3.0× | **3.5×** |
+| **VQeval composite** (Dense→LVSA) | 63→63 | 59→63 | 61→63 | 61→64 | 59→64 | 59→63 |
+
+The speedup **grows with length** (LVSA is the dense regime at 1× by design, so
+the win starts at 2×), and quality *improves* at extension — LVSA's rotating
+keyframes prevent the looping/static output dense produces beyond its horizon
+(VQeval loop-quality +27 to +34 at ≥3×). At long horizons dense **OOMs on 80 GB**
+(HunyuanVideo ≥2×, Wan2.1-14B and Cosmos at 6×) where LVSA still generates.
+
+→ Full sweep (5 models × 6 horizons, VQeval + VBench-Long) and the
+quality-metric methodology:
+**[benchmarks/](https://github.com/JiusiServe/LongVideoSparseAttention/tree/main/benchmarks)**.
+
+## Quickstart
+
+```bash
+# 1. vLLM-Omni 0.22.0 (stable) + matching vLLM
+pip install "vllm==0.22.0"
+pip install --no-build-isolation \
+  "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@v0.22.0"
+
+# 2. LVSA core + plugin
+git clone https://github.com/JiusiServe/LongVideoSparseAttention.git
+( cd LongVideoSparseAttention && pip install -e . && pip install -e lvsa-vllm-omni/ )
+
+# 3. this showcase's client dep
+pip install requests
+```
+
+FlashInfer (used by the default `flashinfer` backend) **ships with vLLM** — step 1
+already pulls the pinned `flashinfer-python` + `flashinfer-cubin`, so the fused
+backend works with no extra install. Two optional extras:
+
+```bash
+# (optional) faster startup — prebuilt JIT cache, matched to your CUDA + flashinfer
+CU=cu128   # match your CUDA: cu121 / cu124 / cu128 / …
+FIVER=$(python -c "import importlib.metadata as m; print(m.version('flashinfer-python'))")
+pip install --extra-index-url "https://flashinfer.ai/whl/${CU}" \
+  "flashinfer-jit-cache==${FIVER}+${CU}"
+
+# (fallback) no FlashInfer in your environment? run the SDPA backend instead:
+export LVSA_BACKEND=sdpa
+```
+
+```bash
+# Serve an LVSA-enabled endpoint (Wan 2.1 1.3B at 2× horizon):
+MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers MODEL_FAMILY=wan FRAMES=161 \
+  bash serve_lvsa.sh
+
+# In another shell — generate:
+python generate.py --frames 161 --height 480 --width 832 \
+  --prompt "A dog running through a sunlit forest." --out wan_lvsa.mp4
+```
+
+`serve_lvsa.sh` sets the LVSA env vars for the chosen `MODEL_FAMILY`
+(`wan` / `hunyuan`). Wan and HunyuanVideo run through the LVSA **attention
+backend** by default — it stays sparse even under sequence-parallel, unlike the
+monkey-patch hooks.
+
+## Configuration
+
+The one knob you must get right is the model's training horizon
+(`LVSA_REFERENCE_LATENT_FRAMES`: Wan = 21, HunyuanVideo = 33); `serve_lvsa.sh`
+sets it per family. For the full environment-variable reference, the multi-GPU
+support matrix, and tuning (`sparsity_scale`, rotating keyframes, non-standard
+resolutions), see the upstream docs:
+
+- **[lvsa-vllm-omni plugin reference](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/lvsa-vllm-omni/README.md)** — every `LVSA_*` env var
+- **[docs/parallelism.md](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/docs/parallelism.md)** — TP / Ulysses / HSDP support per model
+- **[docs/tuning.md](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/docs/tuning.md)** — picking knobs for your model and horizon
@@ -0,0 +1,93 @@
+#!/usr/bin/env python
+"""Generate a long video through an LVSA-enabled vLLM-Omni server.
+
+A thin client over the Omni video API — start the server first with
+``serve_lvsa.sh``. The sparse attention runs server-side; this just submits a
+job and saves the result.
+
+Uses the **asynchronous** endpoint (``POST /v1/videos`` → poll
+``GET /v1/videos/{id}`` → download ``GET /v1/videos/{id}/content``) rather than
+``POST /v1/videos/sync``. The sync endpoint is capped by a fixed server-side
+timeout (``VIDEO_SYNC_TIMEOUT_S``, 600 s) and is documented as
+"for benchmark and testing scenarios" — long videos (LVSA's use case) routinely
+exceed it. The async job has no such cap.
+
+  python generate.py --frames 481 --height 480 --width 832 \
+      --prompt "A golden retriever running through a sunlit forest." \
+      --out dog_forest_lvsa.mp4
+"""
+
+import argparse
+import os
+import time
+
+import requests
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--server", default=os.environ.get("OMNI_SERVER", "http://localhost:8000"))
+    ap.add_argument("--prompt", default="A dog running through a sunlit forest.")
+    ap.add_argument("--frames", type=int, default=161)  # Wan 2x horizon
+    ap.add_argument("--height", type=int, default=480)
+    ap.add_argument("--width", type=int, default=832)
+    ap.add_argument("--fps", type=int, default=16)
+    ap.add_argument("--steps", type=int, default=50)
+    ap.add_argument("--guidance", type=float, default=5.0)
+    ap.add_argument("--out", default="lvsa_long_video.mp4")
+    ap.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between status polls.")
+    ap.add_argument(
+        "--max-wait", type=float, default=7200.0, help="Give up if the job hasn't finished after this many seconds."
+    )
+    args = ap.parse_args()
+
+    base = args.server.rstrip("/")
+    form = {
+        "prompt": args.prompt,
+        "size": f"{args.width}x{args.height}",
+        "num_frames": str(args.frames),
+        "fps": str(args.fps),
+        "num_inference_steps": str(args.steps),
+        "guidance_scale": str(args.guidance),
+    }
+
+    # ── 1. submit the async job ──────────────────────────────────────────────
+    print(f"[lvsa-app] POST {base}/v1/videos  {args.width}x{args.height} {args.frames}f steps={args.steps}")
+    resp = requests.post(f"{base}/v1/videos", data=form, timeout=120)
+    resp.raise_for_status()
+    job = resp.json()
+    vid = job["id"]
+    print(f"[lvsa-app] job {vid} ({job.get('status', '?')})")
+
+    # ── 2. poll until completed / failed ─────────────────────────────────────
+    deadline = time.time() + args.max_wait
+    last = None
+    while True:
+        if time.time() > deadline:
+            raise SystemExit(f"[lvsa-app] timed out after {args.max_wait:.0f}s (job {vid} still running)")
+        time.sleep(args.poll_interval)
+        r = requests.get(f"{base}/v1/videos/{vid}", timeout=60)
+        if r.status_code != 200:
+            # retrieve_video returns a non-200 JSON when the job has FAILED
+            raise SystemExit(f"[lvsa-app] job failed: {r.text}")
+        st = r.json()
+        status = st.get("status")
+        progress = st.get("progress", 0)
+        if (status, progress) != last:
+            print(f"[lvsa-app] {status}  {progress}%")
+            last = (status, progress)
+        if status == "completed":
+            break
+        if status == "failed":
+            raise SystemExit(f"[lvsa-app] job failed: {st.get('error')}")
+
+    # ── 3. download the rendered mp4 ─────────────────────────────────────────
+    c = requests.get(f"{base}/v1/videos/{vid}/content", timeout=600)
+    c.raise_for_status()
+    with open(args.out, "wb") as f:
+        f.write(c.content)
+    print(f"[lvsa-app] saved {args.out}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Start an LVSA-enabled vLLM-Omni server for Wan or HunyuanVideo.
+#
+# LVSA is selected as the attention backend via --diffusion-attention-config (see
+# below); the `LVSA_*` env vars only tune it. The `lvsa-vllm-omni` plugin registers
+# itself through vLLM-Omni's `general_plugins` entry points — nothing in vLLM-Omni
+# core is modified. Watch the log for "[LVSA] Geometry detected" to confirm sparse
+# attention is active for your request geometry.
+#
+#   MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers MODEL_FAMILY=wan     FRAMES=161 bash serve_lvsa.sh
+#   MODEL=/path/to/HunyuanVideo-1.5-...       MODEL_FAMILY=hunyuan FRAMES=193 bash serve_lvsa.sh
+set -euo pipefail
+
+MODEL=${MODEL:?set MODEL=/path/to/Wan-or-HunyuanVideo-Diffusers}
+MODEL_FAMILY=${MODEL_FAMILY:-wan}     # wan | hunyuan
+PORT=${PORT:-8000}
+FRAMES=${FRAMES:-161}                 # target horizon in frames
+VAE_T=${VAE_T:-4}                     # VAE temporal compression (Wan/Hunyuan = 4)
+BACKEND=${LVSA_BACKEND:-flashinfer}   # flashinfer (fused, fastest) | sdpa
+
+T_LAT=$(( (FRAMES - 1) / VAE_T + 1 ))
+
+# Per-family training reference horizon (Wan train ref = 21, Hunyuan = 33).
+case "$MODEL_FAMILY" in
+  wan)     REFERENCE=${REFERENCE:-21} ;;
+  hunyuan) REFERENCE=${REFERENCE:-33} ;;
+  *) echo "MODEL_FAMILY must be 'wan' or 'hunyuan'"; exit 1 ;;
+esac
+export LVSA_BACKEND="$BACKEND"
+export LVSA_TOTAL_LATENT_FRAMES="$T_LAT"
+export LVSA_REFERENCE_LATENT_FRAMES="$REFERENCE"
+export LVSA_ROTATE_KEYFRAMES=${LVSA_ROTATE_KEYFRAMES:-1}
+
+echo "[lvsa-app] serving $MODEL ($MODEL_FAMILY) on :$PORT"
+echo "[lvsa-app]   LVSA backend=$BACKEND  T_lat=$T_LAT (frames=$FRAMES)  reference=$REFERENCE"
+echo "[lvsa-app]   confirm activation by grep '\\[LVSA\\] Geometry detected' in this log"
+
+# vLLM-Omni 0.22 selects the LVSA attention backend per-role via
+# --diffusion-attention-config. The backend is what drives sparse attention
+# here; it engages under tensor-parallel too, unlike the LVSA_*_HOOK monkey-patch
+# path (which falls back to dense under sequence-parallel), so we deliberately do
+# NOT set a hook.
+exec vllm serve "$MODEL" --omni --host 0.0.0.0 --port "$PORT" \
+     --diffusion-attention-config '{"per_role": {"self": {"backend": "LVSA"}}}' \
+     --init-timeout 1800 ${EXTRA_ARGS:-}