vllm-omni/docs/serving/videos_api.md

Videos API

vLLM-Omni provides an OpenAI-compatible video generation API for diffusion video models. The API supports asynchronous video jobs through /v1/videos and a synchronous benchmark-oriented endpoint through /v1/videos/sync.

Each server instance runs a single model specified at startup with vllm serve <model> --omni.

Quick Start

Start the Server

vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091

Create a Video Job

create_response=$(curl -s http://localhost:8091/v1/videos \
  -F "prompt=A cinematic tracking shot of a mountain lake at sunrise" \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16" \
  -F "num_inference_steps=40")

video_id=$(echo "${create_response}" | jq -r '.id')

Poll and Download

curl -s "http://localhost:8091/v1/videos/${video_id}" | jq .
curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4

API Reference

Endpoints

Endpoint	Method	Description
/v1/videos	POST	Create an asynchronous video generation job
/v1/videos/sync	POST	Generate a video synchronously and return raw video bytes
/v1/videos/{video_id}	GET	Retrieve job status and metadata
/v1/videos	GET	List stored video jobs
/v1/videos/{video_id}/content	GET	Download generated video content
/v1/videos/{video_id}	DELETE	Delete a video job and stored output

Request Parameters

POST /v1/videos and POST /v1/videos/sync accept multipart/form-data.

OpenAI-style fields

Parameter	Type	Default	Description
prompt	string	required	Text prompt for video generation
model	string	server's model	Optional model name
seconds	string	null	Requested clip duration in seconds
size	string	null	Requested output size in WIDTHxHEIGHT format
user	string	null	Optional user identifier

vLLM-Omni extension fields

Parameter	Type	Default	Description
input_reference	file	null	Uploaded reference image or video for image-to-video/video-to-video requests
control_reference	file	null	Optional uploaded image/video control, up to 512 MiB, for models that declare control-upload support
control_type	string	null	Model control name associated with control_reference; currently Cosmos3 supports edge, blur, depth, seg, and wsm
image_reference	string	null	JSON-encoded reference image payload; do not combine with input_reference or video_reference
video_reference	string	null	JSON-encoded reference video payload; do not combine with input_reference or image_reference
audio_reference	string	null	JSON-encoded audio reference for speech-to-video: {"audio_url": "..."} — supports HTTP(s) URLs or base64 data URLs
source_video	file	null	MiniMax H3 latent-edit source video (.mp4 or .mov, up to 512 MiB)
source_audio	file	null	Optional MiniMax H3 latent-edit source audio (.wav or .mp3, up to 512 MiB)
video_noise_mask	file	null	UTF-8 JSON MiniMax H3 video mask; 0 preserves and 1 regenerates a token
audio_noise_mask	file	null	UTF-8 JSON MiniMax H3 audio mask; 0 preserves and 1 regenerates a token
width	integer	model default	Output video width
height	integer	model default	Output video height
num_frames	integer	1	Number of generated frames
fps	integer	model default	Output frames per second
num_inference_steps	integer	model default	Number of diffusion steps
guidance_scale	number	null	CFG guidance scale for the low-noise stage
guidance_scale_2	number	null	CFG guidance scale for the high-noise stage
boundary_ratio	number	null	Boundary split ratio for multi-stage denoising
flow_shift	number	null	Scheduler flow-shift value
true_cfg_scale	number	null	True CFG scale when supported by the model
seed	integer	null	Random seed for reproducibility
generate_sound	boolean	false	Request model-generated audio for video models that support sound generation
sound_duration	number	null	Duration in seconds for generated audio; defaults to generated video duration
negative_prompt	string	null	Text describing what to avoid in the generated video
enable_frame_interpolation	boolean	null	Enable post-generation frame interpolation
frame_interpolation_exp	integer	null	Interpolation exponent; 1=2x, 2=4x, and so on
frame_interpolation_scale	number	null	RIFE inference scale
frame_interpolation_model_path	string	null	Local path or Hugging Face repo for the interpolation model
lora	string	null	JSON-encoded LoRA configuration object
extra_params	string	null	JSON-encoded object for additional model-specific parameters

Create Response

POST /v1/videos returns a job record:

{
  "id": "video-123",
  "status": "queued",
  "created_at": 1701234567
}

The final content is available from /v1/videos/{video_id}/content after the job status becomes completed.

queued means the request is still waiting for diffusion scheduler admission. The status changes to in_progress when the scheduler first selects the request for execution.

DELETE /v1/videos/{video_id} issues a bounded engine abort (VLLM_OMNI_ABORT_TIMEOUT, default 2s), then cancels the frontend task. Cancellation cleanup is also bounded and best-effort: it confirms the abort was submitted. In request execution mode, MiniMax-H3 checks cancellation at model boundaries, including before and after input preparation, after each denoising step, and before decode, and skips the remaining generation when cancelled. Input preparation or decoding already in progress may finish before the next boundary. Successful steps do not force device synchronization for cancellation. Parallel ranks agree before stopping; independent requests sharing a distributed AllGather offload wave can stop that wave early only when all its requests are cancelled, so a cancelled request cannot strand its live peers in a collective. In step execution mode, cancellation is handled by the scheduler between steps; the request-mode component-boundary checks do not apply within an active step. Other pipelines may still drain their current request batch. The job is then re-read so a completed save is not orphaned.

Synchronous Response

POST /v1/videos/sync blocks until generation finishes and returns raw video bytes. It is useful for benchmarks and simple scripts that do not need job storage or polling.

Examples

Image-to-Video

curl -s http://localhost:8091/v1/videos \
  -F "prompt=animate this image with subtle camera movement" \
  -F "input_reference=@input.png" \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16"

Video-to-Video

For models that support video conditioning, upload the reference video with input_reference:

curl -s http://localhost:8091/v1/videos \
  -F "prompt=continue this motion with consistent subjects and lighting" \
  -F "input_reference=@input.mp4;type=video/mp4" \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16"

You can also pass a JSON-safe video URL or data:video/...;base64,... payload through video_reference. Do not send video_reference together with input_reference or image_reference.

curl -s http://localhost:8091/v1/videos \
  -F "prompt=continue this motion with consistent subjects and lighting" \
  -F 'video_reference={"video_url":"https://example.com/input.mp4"}' \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16"

JSON references currently support image_url/video_url; file_id references are not implemented yet. Models may expose additional V2V controls through extra_params. For example, Cosmos3 supports condition_frame_indexes_vision and condition_video_keep to select which decoded reference frames are used as clean conditioning. Cosmos3 transfer mode also accepts edge, blur, depth, seg, or wsm control hints. Each hint may specify its own control_path and control_weight. Request-level transfer options include control_guidance, control_guidance_interval, emphasize_control_in_prompt, num_video_frames_per_chunk, num_conditional_frames, show_control_condition, and show_input. Transfer uses its transfer-specific system prompt and, by default, appends a control-adherence directive to the positive prompt. It adds duration/FPS and resolution metadata to both CFG branches but does not add a negative prompt automatically. The Cosmos3 recipe includes an optional reference negative prompt and shows how to pass it. Set emphasize_control_in_prompt, use_duration_template, or use_resolution_template to false to disable the corresponding addition. negative_metadata_mode accepts same, inverse, or none and defaults to same for transfer. See the Cosmos3 recipe for complete examples.

A client that cannot place the control on the server filesystem can upload one control with control_reference and identify it with control_type. The API streams the upload to request-scoped storage, supplies its path to the model, and removes it after synchronous or asynchronous generation completes. Other options for the selected control can remain in extra_params; do not also set control or control_path there. Uploads larger than 512 MiB are rejected.

curl -s http://localhost:8091/v1/videos/sync \
  -F "prompt=Preserve the scene while following the world-state control" \
  -F "input_reference=@input.mp4;type=video/mp4" \
  -F "control_reference=@wsm.mp4;type=video/mp4" \
  -F "control_type=wsm" \
  -F 'extra_params={"wsm":{"control_weight":1.0}}' \
  -o output.mp4

HTTP redirects for image_reference.image_url follow vLLM's VLLM_MEDIA_URL_ALLOW_REDIRECTS setting. Before starting the server, set it to 1 (the default) to allow redirects or 0 to reject them. A redirect target can differ from the original host, and vLLM-Omni does not yet fully implement upstream vLLM's media URL allowlist protection. Deployments should therefore treat remote media URLs as untrusted and choose this setting as part of their URL access policy.

MiniMax H3 Latent-Mask Editing

MiniMax H3 accepts request-scoped source media and video/audio noise masks. At least one mask is required. A nontrivial video_noise_mask requires source_video, while a nontrivial audio_noise_mask requires either source_audio or a source_video with an audio stream. Mask values are in [0, 1]: 0 preserves the source, 1 regenerates it, and fractional values blend the two behaviors. Exact all-one masks are no-ops and do not require a source. Source uploads without a mask are rejected. Masks may be a JSON scalar, an array matching the H3 latent/token grid, or (video only) a frame-space array that the server resizes to the latent grid.

For an aligned output of F frames at W x H, the video latent grid is [Tv, H/16, W/16], where Tv = 2 + 5 * ((F - 5) / 17). The video mask may be a scalar, a flat token vector, [Tv, H/32, W/32], the full latent grid, a spatial [h, w] mask applied to every frame, or a frame-space [T, h, w] mask with one slice per 24 fps output frame. A shape that matches the token or full latent grid is always read as that grid. Frame-space masks are area-resized to [H/16, W/16]; along time, a mask shorter than F is padded with its last slice (as the source video is) and a longer one is truncated, then each latent takes the maximum over the frames the causal video VAE folds into it, so any regenerated frame regenerates its latent. Because the 8 MiB limit below applies to the JSON file, clients should area-downsample frame-space masks by 16 spatially before upload. For the model input, timestep, and velocity, a full-grid mask is max-pooled over each 2x2 spatial token and fractional values are rounded upward to 1/256 levels. The final x0 restore uses the original, unquantized mask, so full-grid masks retain cell-level preservation inside a model token. The audio length is Ta = round(F * 40 / 24), and its mask may be a scalar, [Ta], [2, Ta], or a flat 2 * Ta vector. If source audio is short, its missing latent tail is forced to mask value 1 so H3 generates that portion. Each mask must be sent as a UTF-8 JSON file part and is limited to 8 MiB. A file may contain either a JSON scalar or an array.

curl -s http://localhost:8091/v1/videos/sync \
  -F "prompt=Partially restyle the complete clip and soundtrack" \
  -F 'extra_params={"task":"t2va","duration":4.0,"aspect_ratio":"16:9"}' \
  -F "source_video=@source.mp4;type=video/mp4" \
  -F "source_audio=@source.wav;type=audio/wav" \
  -F "video_noise_mask=@video-mask.json;type=application/json" \
  -F "audio_noise_mask=@audio-mask.json;type=application/json" \
  -o edited.mp4

For example, each mask file in the request above may contain the JSON scalar 0.5.

The server streams source files to temporary request-scoped storage and removes them after synchronous or asynchronous generation finishes. Models that do not declare latent-mask editing support reject these fields.

Speech-to-Video

For models that support audio-driven generation (e.g., Wan2.2-S2V), pass both an image reference and an audio reference. The audio_reference field accepts a JSON string with audio_url pointing to an HTTP(s) URL or base64 data URL.

curl -s http://localhost:8091/v1/videos \
  -F "prompt=A person singing" \
  -F 'image_reference={"image_url": "https://example.com/face.png"}' \
  -F 'audio_reference={"audio_url": "https://example.com/speech.mp3"}' \
  -F "width=832" \
  -F "height=480" \
  -F "num_inference_steps=40" \
  -F "guidance_scale=4.5" \
  -F "fps=16"

Synchronous Generation

curl -X POST http://localhost:8091/v1/videos/sync \
  -F "prompt=A small robot walking through a neon city" \
  -F "width=854" \
  -F "height=480" \
  -F "num_frames=80" \
  -F "fps=16" \
  -o output.mp4

Output Encoding

These extra_params control how the server turns decoded frames into MP4 bytes.

Field	Type	Default	Description
preencode_mp4	boolean	false	Encode the MP4 on the worker while the VAE is still decoding, instead of after the full video is materialized
preencode_batch_frames	positive integer	17 (H3, Wan T2V/I2V); 1 (Wan S2V)	Minimum accumulated frames per worker transfer/encoding batch; used only with preencode_mp4=true
video_codec_options	object	null	Encoder options passed through to the H.264 encoder, such as {"preset": "ultrafast", "threads": "0"}

With preencode_mp4 enabled, each committed VAE chunk leaves the accelerator and is encoded while later chunks are still decoding, so host transfer and CPU encoding overlap the remaining decode instead of following it. The response is unchanged: the same complete MP4, byte-for-byte equivalent frames.

curl -X POST http://localhost:8091/v1/videos/sync \
  -F "prompt=A small robot walking through a neon city" \
  -F 'extra_params={"preencode_mp4": true, "preencode_batch_frames": 33, "video_codec_options": {"preset": "ultrafast"}}' \
  -o output.mp4

Set preencode_batch_frames in extra_params (or extra_args for offline sampling) to tune batching. The worker accumulates complete VAE chunks until it has at least this many frames, then transfers and encodes them together. It always flushes the final partial batch. This is a threshold, not an exact chunk length: a value of 1 submits every native chunk immediately, and a value smaller than a native chunk does not split it. Larger values reduce transfers but retain more frames on the accelerator and delay encoding. The VAE decode window and output frame count stay unchanged. Wan S2V keeps its existing per-clip behavior by default. Zero, negative, fractional, boolean, string, and null values are rejected when pre-encoding is enabled.

The transfer ring applies backpressure before another D2H copy when both of its two slots are occupied or the combined pending uint8 payload would exceed 256 MiB. This byte bound keeps large resolutions and uneven final chunks from turning a fixed item count into unexpectedly large memory growth. A native VAE chunk larger than 256 MiB is admitted only when the ring is otherwise empty, so supported chunk shapes cannot deadlock.

preencode_mp4 applies to the complete-MP4 response paths only. The /v1/realtime/video WebSocket endpoint rejects it, because that path already overlaps encoding through its own incremental fragmented-MP4 encoder. Wan also rejects it together with enable_frame_interpolation, which needs the decoded frames the pre-encoded path no longer materializes.

Support is per model: MiniMax-H3 and Wan 2.2 (T2V, I2V, and S2V) implement it, and other models ignore the flag and take the full-decode path.

Storage

Set VLLM_OMNI_SERVER_STORAGE__PATH to control where asynchronous video outputs are stored:

export VLLM_OMNI_SERVER_STORAGE__PATH=/var/tmp/vllm-omni-videos

VLLM_OMNI_STORAGE_PATH is deprecated and will be removed in a future release; use VLLM_OMNI_SERVER_STORAGE__PATH instead.

Model-Specific Examples

For complete text-to-video, image-to-video, and model-specific video-to-video walkthroughs, see:

• Text-to-Video
• Image-to-Video
• Speech-to-Video for Wan2.2-S2V audio-driven lip-sync generation
• Cosmos3 recipes for model-specific video-to-video examples and conditioning controls