vllm-omni/recipes/ByteDance/SeedVR2-RTX-5090.md

SeedVR2 video restoration — RTX 5090

Summary

• Vendor: ByteDance
• Model: SeedVR2 3B
• Task: restore or upscale an uploaded video
• Mode: native offline/HTTP pipeline, one Euler step
• Hardware: NVIDIA GeForce RTX 5090, 32 GiB per device
• Maintainer: 0z5a

When to use this recipe

Run short, constant-frame-rate videos with the native SeedVR2 pipeline. This recipe covers the whole-clip path and the separate long-video route. Color correction from external applications is a separate execution semantic.

Supported model contract

Property	Contract
Model directory	See checkpoint files and hashes
Input	One uploaded video; blank prompt; positive dimensions divisible by 16
Output	MP4 at requested geometry, original frame count and source frame rate
Audio	First mono/stereo track, aligned by PTS; re-encoded as AAC
Sampling	One Euler step, CFG=1, per-request seed
Precision	3B DiT and VAE FP16
Parallelism	Whole-window SP via ulysses_degree; replicated model weights
Frame padding	Internal 4n+1 padding, cropped back; five returns five, six returns six

References

• Canonical SeedVR
• Model guide
• Integration RFC

Hardware

RTX 5090 with 32 GiB per GPU, PCIe, one GPU for the default command or two/four for SP. Each GPU must fit a complete model copy and its activations. Host RAM must accommodate checkpoint staging per rank. Admission uses a padded five-frame 848×480 output pixel budget by default, allowing longer clips at smaller resolutions, plus a 257-frame decoder-work cap. The validated SP4 configuration with VAE tiling and vae_patch_parallel_size=4 uses a padded five-frame 2560×1472 budget, as does any higher matched degree. On devices with more memory the clip budget grows automatically; the other caps are raised through VLLM_OMNI_SEEDVR2_SHARDED_FRAME_PIXELS and VLLM_OMNI_SEEDVR2_MAX_FRAMES, and VLLM_OMNI_SEEDVR2_SHARDED_CLIP_PIXELS overrides the clip budget. The model guide lists exact bounds and which profiles have completed GPU validation.

Software environment

Linux x86-64, Python 3.12, PyTorch 2.13.0+cu130, CUDA 13.0, vLLM 0.29.0, and PyAV in the existing environment. Use the vLLM-Omni branch containing this recipe.

Command

Set MODEL_DIR to the authorized checkpoint directory documented in the model guide; it must also contain model_index.json selecting SeedVR2Pipeline.

CUDA_VISIBLE_DEVICES=0 vllm serve "$MODEL_DIR" --omni \
  --model-class-name SeedVR2Pipeline --dtype float16 --enforce-eager \
  --num-gpus 1 --host 127.0.0.1 --port 8098

For SP=2, expose two GPUs and replace the GPU count with:

--num-gpus 2 --distributed-executor-backend mp \
  --stage-overrides '{"0":{"ulysses_degree":2}}'

Use degree 4 and four visible GPUs for SP=4. Combine any feature-specific settings below into the same stage-overrides object; do not repeat that flag. The validated 16-frame 720p original-size profile is:

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve "$MODEL_DIR" --omni \
  --model-class-name SeedVR2Pipeline --dtype float16 --enforce-eager \
  --vae-use-tiling --num-gpus 4 --distributed-executor-backend mp \
  --stage-overrides '{"0":{"ulysses_degree":4,"vae_patch_parallel_size":4,"vae_parallel_mode":"spatial_shard_height"}}' \
  --host 127.0.0.1 --port 8098

Use size=1280x720 for a landscape 1280×720 input or size=720x1280 for a portrait 720×1280 input. Both require 16 decoded source frames and retain the source frame rate; no upscaling is requested.

Verification

curl --fail-with-body http://127.0.0.1:8098/v1/videos/sync \
  -F 'prompt= ' -F 'input_references=@input.mp4;type=video/mp4' \
  -F 'size=224x128' -F 'num_inference_steps=1' \
  -F 'guidance_scale=1' -F 'seed=7723' --output restored.mp4
python - <<'PY'
import av
with av.open('restored.mp4') as video:
    stream = video.streams.video[0]
    print(stream.width, stream.height, stream.average_rate,
          sum(1 for _ in video.decode(video=0)), len(video.streams.audio))
PY

Expect HTTP 200 and a completely decodable 224×128 video. Check the frame count, FPS, timestamps, and audio against the input. Repeat the same seed and compare decoded frames; changing the seed should change the restored video. Inspect matching first/middle/last frames at the same display scale. AAC is re-encoded, so input/output audio packet hashes need not be identical.

The released 3B checkpoint also passed these four-GPU SP4 HTTP cases with the configuration above. Each input had 16 frames at 24 FPS; the output retained the original size, 16 decoded frames, timestamps, and decodable audio. All four workers and the server exited normally after each run.

1× input → output	Earlier six-frame admission	Revised HTTP result	Request time
1280×720, 16 frames → 1280×720	Rejected	HTTP 200; media checks passed	3.553 s
720×1280, 16 frames → 720×1280	Rejected	HTTP 200; media checks passed	3.388 s

These are single requests on a shared host, not performance comparisons.

One long-video request

For a 24 FPS source, the long route can restore up to 7,200 frames at 768×1344. Use the SP4/VAE height-sharding command above, serve with --api-server-count 1, and set a persistent VLLM_OMNI_SEEDVR2_LONG_OUTPUT_DIR before starting. If the source is shorter than 300 seconds, loop_input=true repeats its frames and audio. The service runs the longest model windows the clip budget admits, with four-frame overlap, and writes one MP4; it does not concatenate separate video files. Download the result within VLLM_OMNI_SEEDVR2_LONG_JOB_TTL_SECONDS, after which the job directory is swept; DELETE on the job URL stops a run at the next window boundary.

curl --fail-with-body http://127.0.0.1:8098/v1/seedvr2/restore-long \
  -F 'input_references=@input.mp4;type=video/mp4' \
  -F 'prompt= ' -F 'size=768x1344' -F 'num_frames=7200' \
  -F 'loop_input=true' -F 'seed=7723' -F 'extra_params={"color_correction_method":"lab"}'
# Use the returned id to poll /v1/seedvr2/restore-long/{id};
# download /v1/seedvr2/restore-long/{id}/content when completed.

The input must have constant 24 FPS timestamps starting at zero. Only one job can run per API server, and --api-server-count must be one. The existing whole-clip /v1/videos/sync limits remain in force. Full 7,200-frame GPU validation is in progress; use this route only with a memory-checked SP4 deployment.

Set VLLM_TEST_SEEDVR2_MODEL_DIR to the directory containing all model files:

python -m pytest -o addopts='' -v tests/diffusion/models/seedvr2/test_seedvr2_e2e.py

The tests run the complete native pipeline for every colour mode and the HTTP long-video route at USP 1 and 8 (eight GPUs required for the latter). The HTTP cases force a 13-frame window, restore 20 frames across a seam, preserve audio and timestamps, reject invalid extra_params, and cancel a longer job. They use the released model rather than mocking its forward pass.

Supported features

Feature	Status
Window SP	Model-local regular/shifted window attention
CPU offload, LoRA, compiled execution, CFG/TP/PP	Unsupported
VFR or multichannel audio	Unsupported
VAE temporal/spatial tiling	Available with --vae-use-tiling
Colour correction	On by default; color_correction_method=lab|wavelet|adain|none
Quantization	Separate opt-in feature

Measurement scope

Compare identical inputs, seeds, dimensions, GPU counts, and warmups. Separate correctness from performance: use repeated interleaved HTTP measurements before claiming a stable gain. Keep raw logs, tensors, and generated media outside the source diff; link selected visuals from the PR's test results.

Temporal tiling and VAE patch parallelism

Use --vae-use-tiling for causal temporal chunks. On a single GPU this also bounds large convolution workspace with exact spatial halos. To shard VAE activations across the same two SP ranks, use:

--vae-use-tiling --num-gpus 2 --distributed-executor-backend mp \
  --stage-overrides '{"0":{"ulysses_degree":2,"vae_patch_parallel_size":2,"vae_parallel_mode":"spatial_shard_height"}}'

The VAE patch degree must equal the SP degree. Height sharding is supported; width sharding and batch slicing are not. Compare clips that cross a temporal chunk boundary as well as five/six-frame clips, and inspect frames adjacent to chunk boundaries. Reduced peak memory does not by itself establish lower latency. High-resolution clips can still exceed device capacity; validate the intended frame count and output size on the target GPUs.