vllm-omni/docs/serving/streaming_video_output_api.md

Streaming Video Output WebSocket API

Use WS /v1/realtime/video to receive fragmented MP4 chunks while a compatible diffusion video model is generating. This endpoint streams generated video output; it is unrelated to WS /v1/video/chat/stream, which streams live video input into a conversational model.

Quick Start

Start a model that supports step execution and enable diffusion streaming output:

vllm serve BestWishYsh/Helios-Distilled --omni \
  --diffusion-streaming-output \
  --port 8091

Install the client dependency and run the provided client:

pip install av websockets

python examples/online_serving/streaming_video_generation/streaming_video_client.py \
  --host 127.0.0.1 \
  --port 8091 \
  --model BestWishYsh/Helios-Distilled \
  --prompt "A serene lakeside sunrise with mist over the water" \
  --output output.mp4

Protocol

Control events are JSON text frames. Each video.chunk_metadata event is immediately followed by one binary WebSocket frame containing fragmented MP4 bytes.

Direction	Event or frame	Purpose
Client to server	session.start	Starts generation with a prompt and video parameters
Server to client	video.start	Confirms the request ID, format, and accepted configuration
Server to client	video.chunk_metadata	Describes the next binary media frame
Server to client	Binary frame	Carries one fragmented MP4 chunk
Client to server	session.interaction	Queues an optional prompt update during generation
Server to client	session.interaction.queued	Confirms that an interaction was queued
Client to server	session.ping	Refreshes the stall clock
Server to client	session.pong	Acknowledges a ping
Client to server	session.stop	Cancels the active generation
Server to client	session.done	Reports completion or cancellation and the chunk count
Server to client	error	Reports validation, generation, or timeout failures

Start a minimal session with:

{
  "type": "session.start",
  "model": "BestWishYsh/Helios-Distilled",
  "prompt": "A serene lakeside sunrise",
  "format": "m4s",
  "width": 640,
  "height": 384,
  "fps": 16,
  "num_frames": 99
}

Only m4s output is currently accepted. Concatenate the binary frames in receive order, then remux the fragmented stream if the target player requires a progressive MP4 file.

Mid-Generation Interaction

Models that implement interaction updates can change the active prompt:

{
  "type": "session.interaction",
  "interaction": {
    "event_id": "sunlight-update",
    "event": {"prompt": "Sunlight breaks through the mist"},
    "transition_chunks": 2
  }
}

This feature is pipeline-dependent. An enabled endpoint does not imply that the loaded model supports midway prompt updates.

The default start timeout is 10 seconds and the server reports a stall after about 60 seconds without generation progress or a session.ping. See the streaming video generation example for prompt-update scheduling and browser playback.