vllm-omni/docs/user_guide/examples/online_serving/speech_to_video.md

Speech-To-Video

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/speech_to_video.

This example demonstrates how to deploy the Wan2.2 speech-to-video (S2V) model for online video generation using vLLM-Omni.

Supported Models

Model	Model ID
Wan2.2 S2V (14B)	Wan-AI/Wan2.2-S2V-14B

Start Server

Basic Start

VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve Wan-AI/Wan2.2-S2V-14B --omni \
  --model-class-name WanS2VPipeline \
  --tensor-parallel-size 2 \
  --flow-shift 3.0 \
  --vae-use-slicing --vae-use-tiling \
  --cache-backend cache_dit \
  --port 8091

Start with Script

bash run_server.sh

The script allows overriding:

• MODEL (default: Wan-AI/Wan2.2-S2V-14B)
• PORT (default: 8091)
• FLOW_SHIFT (default: 3.0)
• TP (default: 2)
• CACHE_BACKEND (default: cache_dit)

Start with 4 GPUs

TP=4 bash run_server.sh

API Endpoints

The server provides OpenAI-compatible video generation endpoints:

• POST /v1/videos: create a video generation job (async)
• POST /v1/videos/sync: generate a video and return raw bytes (sync, for benchmarks)
• GET /v1/videos/{video_id}: retrieve the current job status and metadata
• GET /v1/videos: list stored video jobs
• GET /v1/videos/{video_id}/content: download the generated video file
• DELETE /v1/videos/{video_id}: delete the job and any stored output

Sync API (Recommended for Testing)

POST /v1/videos/sync blocks until generation completes and returns the raw video bytes (video/mp4) directly in the response body.

bash run_curl_speech_to_video.sh

Async Job API

POST /v1/videos creates an asynchronous job and returns immediately.

no_proxy=127.0.0.1 \
create_response=$(curl -sS -X POST http://127.0.0.1:8091/v1/videos \
  -H "Accept: application/json" \
  -F "prompt=A person singing" \
  -F 'image_reference={"image_url": "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"}' \
  -F 'audio_reference={"audio_url": "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"}' \
  -F "width=832" -F "height=480" \
  -F "num_inference_steps=40" \
  -F "guidance_scale=4.5" \
  -F "fps=16")

video_id=$(echo "$create_response" | jq -r '.id')

# Poll until complete
while true; do
  status=$(curl -s "http://127.0.0.1:8091/v1/videos/${video_id}" | jq -r '.status')
  if [ "$status" = "completed" ]; then
    break
  fi
  if [ "$status" = "failed" ]; then
    echo "Video generation failed"
    exit 1
  fi
  sleep 2
done

# Download
curl -L "http://127.0.0.1:8091/v1/videos/${video_id}/content" -o s2v_output.mp4

Request Parameters

Parameter	Type	Default	Description
prompt	str	-	Text description of the desired video
image_reference	JSON str	-	Image reference: {"image_url": "..."} — supports HTTP(s) URLs or base64 data URLs
audio_reference	JSON str	-	Audio reference: {"audio_url": "..."} — supports HTTP(s) URLs or base64 data URLs
width	int	None	Video width in pixels
height	int	None	Video height in pixels
fps	int	None	Output video frame rate
num_inference_steps	int	None	Number of denoising steps (40 recommended)
guidance_scale	float	None	CFG guidance scale
seed	int	None	Random seed for reproducibility

Notes

• S2V requires both a reference image (image_reference) and an audio reference (audio_reference). The generated video will show a person matching the reference image with lip movements synchronized to the audio.
• audio_reference accepts a JSON string: {"audio_url": "..."} where the URL can be:
  • An HTTP/HTTPS URL (e.g., https://example.com/audio.mp3)
  • A base64 data URL (e.g., data:audio/mp3;base64,...)
• --model-class-name WanS2VPipeline is required on the server to select the S2V pipeline (distinct from the T2V/I2V pipelines).
• --cache-backend cache_dit enables DiT caching for ~2x speedup on cached steps.
• Audio is muxed into the output MP4 automatically.
• Always pass fps=16 to ensure correct video/audio alignment.