Enable Wan2.2-S2V modeling to vLLM-omni (#2751)

Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

docs/models/supported_models.md

@@ -32,6 +32,7 @@ th {
 | `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `WanImageToVideoPipeline` | Wan2.2-I2V | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
+| `WanSpeechToVideoPipeline` | Wan2.2-S2V | `Wan-AI/Wan2.2-S2V-14B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `Wan22VACEPipeline` | Wan2.1-VACE | `Wan-AI/Wan2.1-VACE-1.3B-diffusers`, `Wan-AI/Wan2.1-VACE-14B-diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `LTX2Pipeline` | LTX-2-T2V | `Lightricks/LTX-2` | ✅︎ | ✅︎ | | |
 | `LTX2ImageToVideoPipeline` | LTX-2-I2V | `Lightricks/LTX-2` | ✅︎ | ✅︎ | | |

examples/offline_inference/speech_to_video/README.md

@@ -0,0 +1,135 @@
+# Speech-To-Video
+
+This example demonstrates how to generate talking-head videos from a reference image and an audio clip using the Wan2.2 Speech-to-Video (S2V) pipeline with vLLM-Omni's offline inference API.
+
+The S2V pipeline generates multiple video clips autoregressively — the last frames of each clip become the motion context for the next — producing a seamless video that spans the full audio duration.
+
+## Prerequisites
+
+The S2V pipeline requires `librosa` for audio loading:
+
+```bash
+pip install librosa decord
+```
+
+## Local CLI Usage
+
+### Singing Example (480p)
+
+```bash
+wget -O "Five Hundred Miles.png" "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"
+wget -O "Five Hundred Miles.MP3" "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"
+python speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 448 --width 832 \
+  --num-inference-steps 5 \
+  --output s2v_singing_480p.mp4
+```
+
+## Key Arguments
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `--model` | *(required)* | Path to Wan2.2 S2V model (local path or HuggingFace ID) |
+| `--image` | *(required)* | Path to reference image (face/portrait) |
+| `--audio` | *(required)* | Path to audio file (wav/mp3) |
+| `--prompt` | `"A person speaking naturally"` | Text prompt describing the scene |
+| `--negative-prompt` | *(S2V default)* | Negative prompt (built-in Chinese quality filter if not set) |
+| `--height` / `--width` | *auto* | Output resolution (auto-calculated from image aspect ratio if not set, must be divisible by 64) |
+| `--num-frames` | `80` | Number of frames per clip (should be divisible by 4) |
+| `--num-inference-steps` | `40` | Number of denoising steps |
+| `--guidance-scale` | `4.5` | Classifier-free guidance scale |
+| `--flow-shift` | `3.0` | Scheduler flow shift |
+| `--seed` | `42` | Random seed for reproducibility |
+| `--fps` | `16` | Frames per second for the saved MP4 |
+| `--init-first-frame` | `False` | Use reference image as the first frame |
+| `--output` | `s2v_output.mp4` | Output video file path |
+| `--vae-use-slicing` | `False` | Enable VAE slicing for memory optimization |
+| `--vae-use-tiling` | `False` | Enable VAE tiling for memory optimization |
+| `--tensor-parallel-size` | `1` | Number of GPUs for tensor parallelism (TP) inside the DiT |
+| `--cfg-parallel-size` | `1` | Number of GPUs for CFG parallelism (1 or 2) |
+| `--vae-patch-parallel-size` | `1` | Number of GPUs for VAE patch parallelism |
+| `--enable-cpu-offload` | `False` | Enable CPU offloading |
+| `--enable-layerwise-offload` | `False` | Enable layerwise offloading on DiT |
+
+> If you encounter OOM errors, try `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage, or `--enable-cpu-offload` for more aggressive memory saving.
+
+## Performance Tips
+
+The dominant cost in S2V is self-attention across the full token sequence (~80K at 720p), which scales quadratically with resolution. Strategies to speed up generation:
+
+| Strategy | Speedup | How |
+|---|---|---|
+| **Lower resolution** (480p) | ~3.5x | `--height 448 --width 832` — reduces tokens from 80K to 33K |
+| **Tensor parallelism** (2 GPUs) | ~1.4x | `--tensor-parallel-size 2` — splits attention/GEMM across GPUs |
+| **Fewer steps** | Linear | `--num-inference-steps 5` — trades quality for speed |
+| **Combine all above** | ~5x+ | 480p + TP=2 + 5 steps |
+
+## Profiling
+
+To profile the S2V pipeline and identify performance bottlenecks:
+
+### Enable Built-in Profiler
+
+Use `--enable-diffusion-pipeline-profiler` to display stage durations:
+
+```bash
+python speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 448 --width 832 \
+  --num-inference-steps 5 \
+  --enable-diffusion-pipeline-profiler \
+  --output s2v_output.mp4
+```
+
+### Torch Profiler (Device-Level Profiling)
+
+For detailed GPU/XPU kernel-level profiling with Chrome trace visualization:
+
+```bash
+python speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 448 --width 832 \
+  --num-inference-steps 1 \
+  --profile-dir /tmp/s2v_profile \
+  --profile-record-shapes \
+  --profile-with-memory \
+  ----profile-with-stack \
+  --output s2v_profiled.mp4
+```
+
+**Profiler Arguments:**
+
+| Parameter | Description |
+|-----------|-------------|
+| `--profile-dir` | Enable torch profiler and save traces to this directory |
+| `--profile-record-shapes` | Record tensor shapes (increases trace size) |
+| `--profile-with-stack` | Record stack traces (increases overhead) |
+| `--profile-with-memory` | Profile memory usage |
+| `--profile-with-flops` | Estimate FLOPs for operations |
+
+**Output Files:**
+- `trace_rank*.json.gz` - Chrome trace (open in `chrome://tracing`)
+- `ops_rank*.xlsx` - Excel workbook with operation statistics
+
+> **Note:** Use `--num-inference-steps 1` for faster profiling. The trace captures one full denoising step with all transformer blocks and attention operations.
+
+## How It Works
+
+The S2V pipeline processes in the following stages:
+
+1. **Pre-process**: The reference image is resized/cropped to fit the target resolution. The audio path is validated.
+2. **Text encoding**: The prompt is encoded using the UMT5 text encoder.
+3. **Audio encoding**: The audio file is loaded via `librosa`, processed through Wav2Vec2, and bucketed to align with video frame-rate. The audio length determines the number of clips (`num_repeat`).
+4. **Reference image encoding**: The reference image is VAE-encoded into latent tokens.
+5. **Multi-clip denoising**: For each clip, random noise is generated and denoised through the `WanModel_S2V` transformer with audio injection. The last frames of each decoded clip become the motion context for the next.
+6. **Post-process**: All clips are concatenated and converted to pixel-space video frames.

examples/offline_inference/speech_to_video/speech_to_video.py

@@ -0,0 +1,402 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""
+Speech-to-Video generation example using Wan2.2 S2V.
+
+Generates talking-head videos from a reference image and an audio clip
+using the Wan2.2 S2V pipeline with multi-clip autoregressive generation.
+
+Usage:
+    python speech_to_video.py \
+        --model /path/to/Wan2.2-S2V-14B \
+        --image reference.jpg \
+        --audio speech.wav \
+        --prompt "A person speaking naturally"
+"""
+
+import argparse
+import os
+import time
+from pathlib import Path
+
+import torch
+
+from vllm_omni.diffusion.data import DiffusionParallelConfig
+from vllm_omni.entrypoints.omni import Omni
+from vllm_omni.inputs.data import OmniDiffusionSamplingParams
+from vllm_omni.outputs import OmniRequestOutput
+from vllm_omni.platforms import current_omni_platform
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Generate a talking-head video from a reference image and audio (Wan2.2 S2V)."
+    )
+    parser.add_argument(
+        "--model",
+        required=True,
+        help="Path to Wan2.2 S2V model (local path or HuggingFace ID).",
+    )
+    parser.add_argument("--image", required=True, help="Path to reference image.")
+    parser.add_argument("--audio", required=True, help="Path to audio file (wav/mp3).")
+    parser.add_argument(
+        "--prompt",
+        default="A person speaking naturally",
+        help="Text prompt describing the scene.",
+    )
+    parser.add_argument(
+        "--negative-prompt",
+        default=None,
+        help="Negative prompt (uses S2V default if not set).",
+    )
+    parser.add_argument("--seed", type=int, default=42, help="Random seed.")
+    parser.add_argument(
+        "--guidance-scale",
+        type=float,
+        default=4.5,
+        help="CFG scale (default: 4.5).",
+    )
+    parser.add_argument(
+        "--height",
+        type=int,
+        default=None,
+        help="Video height (auto-calculated from reference image if not set).",
+    )
+    parser.add_argument(
+        "--width",
+        type=int,
+        default=None,
+        help="Video width (auto-calculated from reference image if not set).",
+    )
+    parser.add_argument(
+        "--num-frames",
+        type=int,
+        default=80,
+        help="Frames per clip (should be divisible by 4, default: 80).",
+    )
+    parser.add_argument(
+        "--num-inference-steps",
+        type=int,
+        default=40,
+        help="Number of denoising steps (default: 40).",
+    )
+    parser.add_argument(
+        "--flow-shift",
+        type=float,
+        default=3.0,
+        help="Scheduler flow shift (default: 3.0).",
+    )
+    parser.add_argument(
+        "--output",
+        type=str,
+        default="s2v_output.mp4",
+        help="Path to save the output video.",
+    )
+    parser.add_argument(
+        "--fps",
+        type=int,
+        default=16,
+        help="Frames per second for the output video (default: 16).",
+    )
+    parser.add_argument(
+        "--init-first-frame",
+        action="store_true",
+        help="Use the reference image as the first frame of the video.",
+    )
+    parser.add_argument(
+        "--vae-use-slicing",
+        action="store_true",
+        help="Enable VAE slicing for memory optimization.",
+    )
+    parser.add_argument(
+        "--vae-use-tiling",
+        action="store_true",
+        help="Enable VAE tiling for memory optimization.",
+    )
+    parser.add_argument(
+        "--enable-cpu-offload",
+        action="store_true",
+        help="Enable CPU offloading for diffusion models.",
+    )
+    parser.add_argument(
+        "--enable-layerwise-offload",
+        action="store_true",
+        help="Enable layerwise (blockwise) offloading on DiT modules.",
+    )
+    parser.add_argument(
+        "--tensor-parallel-size",
+        type=int,
+        default=1,
+        help="Number of GPUs used for tensor parallelism (TP) inside the DiT.",
+    )
+    parser.add_argument(
+        "--cfg-parallel-size",
+        type=int,
+        default=1,
+        choices=[1, 2],
+        help="Number of GPUs used for classifier free guidance parallel size.",
+    )
+    parser.add_argument(
+        "--vae-patch-parallel-size",
+        type=int,
+        default=1,
+        help="Number of GPUs used for VAE patch/tile parallelism (decode).",
+    )
+    parser.add_argument(
+        "--ring-degree",
+        type=int,
+        default=1,
+        help="Number of GPUs used for ring sequence parallelism.",
+    )
+    parser.add_argument(
+        "--ulysses-degree",
+        type=int,
+        default=1,
+        help="Number of GPUs used for ulysses sequence parallelism.",
+    )
+    parser.add_argument(
+        "--enforce-eager",
+        action="store_true",
+        help="Disable torch.compile and force eager execution.",
+    )
+    parser.add_argument(
+        "--enable-diffusion-pipeline-profiler",
+        action="store_true",
+        help="Enable diffusion pipeline profiler to display stage durations.",
+    )
+    parser.add_argument(
+        "--cache-backend",
+        type=str,
+        default=None,
+        choices=["cache_dit"],
+        help="Cache backend for acceleration. Default: None.",
+    )
+    parser.add_argument(
+        "--enable-cache-dit-summary",
+        action="store_true",
+        help="Enable cache-dit summary logging after diffusion forward passes.",
+    )
+    parser.add_argument(
+        "--profile-dir",
+        type=str,
+        default=None,
+        help="Enable torch profiler and save traces to this directory.",
+    )
+    parser.add_argument(
+        "--profile-record-shapes",
+        action="store_true",
+        help="Record tensor shapes in profiler (increases trace size).",
+    )
+    parser.add_argument(
+        "--profile-with-stack",
+        action="store_true",
+        help="Record stack traces in profiler (increases overhead).",
+    )
+    parser.add_argument(
+        "--profile-with-memory",
+        action="store_true",
+        help="Profile memory usage.",
+    )
+    parser.add_argument(
+        "--profile-with-flops",
+        action="store_true",
+        help="Estimate FLOPs for operations.",
+    )
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
+
+    # Load reference image
+    import PIL.Image
+
+    image = PIL.Image.open(args.image).convert("RGB")
+
+    # Cache-dit config
+    cache_config = None
+    if args.cache_backend == "cache_dit":
+        cache_config = {
+            "Fn_compute_blocks": 1,
+            "Bn_compute_blocks": 0,
+            "max_warmup_steps": 4,
+            "max_cached_steps": 20,
+            "residual_diff_threshold": 0.24,
+            "max_continuous_cached_steps": 3,
+            "enable_taylorseer": False,
+            "taylorseer_order": 1,
+            "scm_steps_mask_policy": None,
+            "scm_steps_policy": "dynamic",
+        }
+
+    parallel_config = DiffusionParallelConfig(
+        tensor_parallel_size=args.tensor_parallel_size,
+        cfg_parallel_size=args.cfg_parallel_size,
+        vae_patch_parallel_size=args.vae_patch_parallel_size,
+        ring_degree=args.ring_degree,
+        ulysses_degree=args.ulysses_degree,
+    )
+
+    # Check if profiling is requested via CLI args or environment variable
+    profile_dir = args.profile_dir or os.getenv("VLLM_TORCH_PROFILER_DIR")
+    profiler_enabled = bool(profile_dir)
+    profiler_config = None
+    if profiler_enabled:
+        from vllm.config import ProfilerConfig
+
+        profiler_config = ProfilerConfig(
+            profiler="torch",
+            torch_profiler_dir=profile_dir,
+            torch_profiler_record_shapes=args.profile_record_shapes,
+            torch_profiler_with_stack=args.profile_with_stack,
+            torch_profiler_with_memory=args.profile_with_memory,
+            torch_profiler_with_flops=args.profile_with_flops,
+        )
+
+    omni = Omni(
+        model=args.model,
+        model_class_name="WanS2VPipeline",
+        flow_shift=args.flow_shift,
+        vae_use_slicing=args.vae_use_slicing,
+        vae_use_tiling=args.vae_use_tiling,
+        enable_cpu_offload=args.enable_cpu_offload,
+        enable_layerwise_offload=args.enable_layerwise_offload,
+        parallel_config=parallel_config,
+        enforce_eager=args.enforce_eager,
+        enable_diffusion_pipeline_profiler=args.enable_diffusion_pipeline_profiler,
+        cache_backend=args.cache_backend,
+        cache_config=cache_config,
+        enable_cache_dit_summary=args.enable_cache_dit_summary,
+        profiler_config=profiler_config,
+    )
+
+    # Print generation configuration
+    print(f"\n{'=' * 60}")
+    print("Generation Configuration:")
+    print(f"  Model: {args.model}")
+    print(f"  Reference image: {args.image}")
+    print(f"  Audio: {args.audio}")
+    print(f"  Inference steps: {args.num_inference_steps}")
+    print(f"  Frames per clip: {args.num_frames}")
+    print(f"  Guidance scale: {args.guidance_scale}")
+    print(f"  Flow shift: {args.flow_shift}")
+    print(f"  Init first frame: {args.init_first_frame}")
+    if args.height and args.width:
+        print(f"  Video size: {args.width}x{args.height}")
+    else:
+        print("  Video size: auto (from reference image)")
+    print(f"{'=' * 60}\n")
+
+    # Start profiling if enabled
+    if profiler_enabled:
+        print("[Profiler] Starting profiling...")
+        omni.start_profile()
+
+    generation_start = time.perf_counter()
+
+    result = omni.generate(
+        {
+            "prompt": args.prompt,
+            "negative_prompt": args.negative_prompt,
+            "multi_modal_data": {
+                "image": image,
+                "audio": args.audio,
+                "init_first_frame": args.init_first_frame,
+            },
+        },
+        OmniDiffusionSamplingParams(
+            height=args.height,
+            width=args.width,
+            num_frames=args.num_frames,
+            num_inference_steps=args.num_inference_steps,
+            guidance_scale=args.guidance_scale,
+            seed=args.seed,
+            generator=generator,
+        ),
+    )
+
+    generation_end = time.perf_counter()
+    generation_time = generation_end - generation_start
+    print(f"Total generation time: {generation_time:.4f} seconds ({generation_time * 1000:.2f} ms)")
+
+    # Stop profiling if enabled
+    if profiler_enabled:
+        print("\n[Profiler] Stopping profiler and collecting results...")
+        profile_results = omni.stop_profile()
+        if profile_results and isinstance(profile_results, list):
+            print("\n" + "=" * 60)
+            print("PROFILING RESULTS:")
+            for i, result in enumerate(profile_results):
+                print(f"\nStage {i}:")
+                if result:
+                    print(f"  {result}")
+            print("=" * 60)
+        else:
+            print("[Profiler] No valid profiling data returned.")
+
+    # Extract output from result
+    output = OmniRequestOutput.unwrap_result(result)
+
+    if not output.images:
+        raise ValueError("No video frames found in OmniRequestOutput.")
+
+    # Extract audio from multimodal_output (set by pipeline post-processor)
+    mm = output.multimodal_output or {}
+    audio_waveform = mm.get("audio")
+    output_fps = float(mm.get("fps", args.fps))
+    output_sr = int(mm.get("audio_sample_rate", 16000))
+
+    if audio_waveform is not None:
+        print(f"Audio waveform: shape={audio_waveform.shape}, sr={output_sr}")
+    else:
+        print("Warning: no audio waveform in pipeline output")
+
+    # Normalize frames to (T, H, W, C) uint8 numpy array
+    import numpy as np
+
+    def _flatten_to_array(data):
+        """Unwrap to a single (T, H, W, C) numpy array."""
+        if isinstance(data, np.ndarray):
+            if data.ndim == 5:
+                return data[0]  # (B, T, H, W, C) → (T, H, W, C)
+            if data.ndim == 4:
+                return data  # already (T, H, W, C)
+        if isinstance(data, list) and data:
+            first_elem = data[0]
+            if isinstance(first_elem, np.ndarray) and first_elem.ndim >= 4:
+                return _flatten_to_array(first_elem)
+            if isinstance(first_elem, np.ndarray) and first_elem.ndim == 3:
+                return np.stack(data)  # list of (H, W, C) → (T, H, W, C)
+        return data
+
+    video_frames = _flatten_to_array(output.images)
+    # postprocess_video returns float32 in [0,1]; mux_video_audio_bytes needs uint8
+    if isinstance(video_frames, np.ndarray) and video_frames.dtype != np.uint8:
+        video_frames = (np.clip(video_frames, 0, 1) * 255).astype(np.uint8)
+    num_frames = video_frames.shape[0] if isinstance(video_frames, np.ndarray) else len(video_frames)
+
+    # Save output video with audio
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    from vllm_omni.diffusion.utils.media_utils import mux_video_audio_bytes
+
+    video_bytes = mux_video_audio_bytes(
+        video_frames,
+        audio_waveform,
+        fps=output_fps,
+        audio_sample_rate=output_sr,
+    )
+    with open(output_path, "wb") as f:
+        f.write(video_bytes)
+
+    print(f"Saved generated video to {output_path}")
+    print(f"Video has {num_frames} frames at {output_fps} fps ({num_frames / output_fps:.1f}s)")
+    if audio_waveform is not None:
+        print(f"Audio: {len(audio_waveform) / output_sr:.1f}s at {output_sr} Hz")
+
+
+if __name__ == "__main__":
+    main()

recipes/Wan-AI/Wan2.2-S2V.md

@@ -0,0 +1,206 @@
+# Wan2.2 Speech To Video
+
+## Summary
+
+- Vendor: Wan-AI
+- Model: `Wan-AI/Wan2.2-S2V-14B`
+- Task: Speech-to-video generation (talking-head / singing from reference image + audio)
+- Mode: Offline inference
+- Maintainer: Community
+
+## When to use this recipe
+
+Use this recipe when you want to generate talking-head or singing videos from a
+reference image and audio clip using the Wan2.2 S2V 14B model with vLLM-Omni.
+Two configurations are provided:
+
+1. **Multi-GPU with tensor parallelism** — splits the DiT across GPUs for faster
+   per-step latency.
+2. **Multi-GPU with tensor parallelism + CPU offload** — enables generation on
+   GPUs with limited VRAM by offloading idle model components to CPU.
+
+## References
+
+- Upstream model card: <https://huggingface.co/Wan-AI/Wan2.2-S2V-14B>
+- Example reference assets: <https://github.com/Wan-Video/Wan2.2/tree/main/examples>
+
+## Hardware Support
+
+## CUDA
+
+### 2× NVIDIA A100/H100 (80 GB)
+
+#### Environment
+
+- OS: Linux
+- Python: 3.10+
+- Driver: NVIDIA driver with CUDA 12.x
+- vLLM version: Match the repository requirements for your checkout
+- vLLM-Omni version or commit: Use the commit you are deploying from
+
+#### Prerequisites
+
+None
+
+
+#### Command
+
+```bash
+wget -O "Five Hundred Miles.png" \
+  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"
+wget -O "Five Hundred Miles.MP3" \
+  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"
+```
+
+**TP=2 (no CPU offload, requires ~60 GB VRAM per GPU):**
+
+```bash
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+python examples/offline_inference/speech_to_video/speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 720 --width 1280 --num-frames 81 \
+  --num-inference-steps 40 \
+  --fps 16 \
+  --tensor-parallel-size 2 \
+  --vae-use-slicing --vae-use-tiling \
+  --output s2v_720p_tp2.mp4
+```
+
+**TP=1 with CPU offload (reduces VRAM usage, adds ~12s overhead):**
+
+```bash
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+python examples/offline_inference/speech_to_video/speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 720 --width 1280 --num-frames 81 \
+  --num-inference-steps 40 \
+  --fps 16 \
+  --tensor-parallel-size 1 \
+  --enable-cpu-offload \
+  --vae-use-slicing --vae-use-tiling \
+  --output s2v_720p_tp1_offload.mp4
+```
+
+**TP=1 * cfg=2 with CPU offload:**
+
+```bash
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+python examples/offline_inference/speech_to_video/speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 720 --width 1280 --num-frames 81 \
+  --num-inference-steps 40 \
+  --fps 16 \
+  --tensor-parallel-size 1 \
+  --enable-cpu-offload \
+  --cfg-parallel-size 2 \
+  --vae-use-slicing --vae-use-tiling \
+  --output s2v_720p_tp1_cfg2_offload.mp4
+```
+
+## XPU
+
+### 4x or 8x Intel Arc B70 (32 GB)
+
+#### Environment
+
+- OS: Linux
+- Python: 3.10+
+- torch: 2.11.0+xpu
+- triton-xpu: 3.7.0
+- vLLM version: Match the repository requirements for your checkout
+- vLLM-Omni version or commit: Use the commit you are deploying from
+
+#### Prerequisites
+
+None
+
+
+#### Command
+
+```bash
+wget -O "Five Hundred Miles.png" \
+  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"
+wget -O "Five Hundred Miles.MP3" \
+  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"
+```
+
+**TP=4:**
+
+```bash
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+python examples/offline_inference/speech_to_video/speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 720 --width 1280 --num-frames 81 \
+  --num-inference-steps 40 \
+  --fps 16 \
+  --tensor-parallel-size 4 \
+  --vae-use-slicing --vae-use-tiling \
+  --output s2v_720p_tp4.mp4
+```
+
+**TP=4 cfg=2:**
+
+```bash
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+python examples/offline_inference/speech_to_video/speech_to_video.py \
+  --model Wan-AI/Wan2.2-S2V-14B \
+  --image "Five Hundred Miles.png" \
+  --audio "Five Hundred Miles.MP3" \
+  --prompt "A person singing" \
+  --height 720 --width 1280 --num-frames 81 \
+  --num-inference-steps 40 \
+  --fps 16 \
+  --tensor-parallel-size 4 \
+  --cfg-parallel-size 2 \
+  --vae-use-slicing --vae-use-tiling \
+  --output s2v_720p_tp4_cfg2_offload.mp4
+```
+
+
+#### Verification
+
+The output video should show a talking/singing person matching the reference
+image with lip movements synchronized to the audio. Check:
+
+1. Video file is generated at the specified output path.
+2. Audio is muxed into the output MP4.
+3. Lip sync quality is reasonable for the given inference steps.
+
+#### Notes
+
+- **Key flags:**
+  - `--tensor-parallel-size <N>` — splits the DiT across N GPUs for TP.
+  - `--cfg-parallel-size <N>` - splits the conditioned / unconditioned pred to 2 GPUs run parallel.
+  - `--enable-cpu-offload` — enables model-level CPU offloading (transformer
+    and text_encoder alternate on GPU). Adds ~12s overhead but reduces VRAM.
+  - `--vae-use-slicing --vae-use-tiling` — reduces VAE peak memory by
+    processing frames in slices and spatial tiles.
+  - `--num-frames 81` — generates ~5s of video at 16 fps.
+  - `--num-inference-steps` — quality/speed tradeoff. 4 steps for fast preview,
+    40 steps for production quality.
+- **Performance tips:**
+  - Use `--enforce-eager` to skip torch.compile if you encounter recompilation
+    issues or want deterministic profiling.
+  - For profiling, add `--enable-diffusion-pipeline-profiler` for stage-level
+    timing, or `--profiler-config '{"profiler":"torch","torch_profiler_dir":"./perf","torch_profiler_record_shapes":true,"torch_profiler_with_stack":false}'`
+    for detailed op-level torch profiler traces.
+  - The first clip in multi-clip generation includes warmup overhead; subsequent
+    clips run at steady-state speed.
+- **Known limitations:**
+  - `VLLM_WORKER_MULTIPROC_METHOD=spawn` is required for multi-GPU offline
+    inference to avoid CUDA context issues with forked processes.
+  - CPU offload adds ~12s latency for model-level shuttling between CPU and GPU.
+  - S2V self-attention is ~8.5× more expensive than T2V due to audio-visual
+    conditioning fused into self-attn blocks.

vllm_omni/diffusion/data.py

@@ -761,6 +761,11 @@ class OmniDiffusionConfig:
                         self.model_class_name = "NextStep11Pipeline"
                     self.tf_model_config = TransformerConfig()
                     self.update_multimodal_support()
+                elif model_type == "s2v":
+                    if self.model_class_name is None:
+                        self.model_class_name = "WanS2VPipeline"
+                    self.tf_model_config = TransformerConfig()
+                    self.update_multimodal_support()
                 elif architectures and len(architectures) == 1:
                     self.model_class_name = architectures[0]
                 else:

vllm_omni/diffusion/models/wan2_2/__init__.py

@@ -17,6 +17,11 @@ from .pipeline_wan2_2_i2v import (
     get_wan22_i2v_post_process_func,
     get_wan22_i2v_pre_process_func,
 )
+from .pipeline_wan2_2_s2v import (
+    Wan22S2VPipeline,
+    get_wan22_s2v_post_process_func,
+    get_wan22_s2v_pre_process_func,
+)
 from .pipeline_wan2_2_ti2v import (
     Wan22TI2VPipeline,
     get_wan22_ti2v_post_process_func,
@@ -42,6 +47,9 @@ __all__ = [
     "WanI2VDMD2Pipeline",
     "get_wan22_i2v_post_process_func",
     "get_wan22_i2v_pre_process_func",
+    "Wan22S2VPipeline",
+    "get_wan22_s2v_post_process_func",
+    "get_wan22_s2v_pre_process_func",
     "Wan22TI2VPipeline",
     "get_wan22_ti2v_post_process_func",
     "get_wan22_ti2v_pre_process_func",

vllm_omni/diffusion/registry.py

@@ -115,6 +115,11 @@ _DIFFUSION_MODELS = {
         "pipeline_wan2_2_i2v",
         "Wan22I2VPipeline",
     ),
+    "WanS2VPipeline": (
+        "wan2_2",
+        "pipeline_wan2_2_s2v",
+        "Wan22S2VPipeline",
+    ),
     "WanT2VDMD2Pipeline": (
         "wan2_2",
         "pipeline_wan2_2",
@@ -434,6 +439,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
     "StableAudioPipeline": "get_stable_audio_post_process_func",
     "AudioXPipeline": "get_audiox_post_process_func",
     "WanImageToVideoPipeline": "get_wan22_i2v_post_process_func",
+    "WanS2VPipeline": "get_wan22_s2v_post_process_func",
     "WanT2VDMD2Pipeline": "get_wan22_post_process_func",
     "WanI2VDMD2Pipeline": "get_wan22_i2v_post_process_func",
     "LongCatImagePipeline": "get_longcat_image_post_process_func",
@@ -469,6 +475,7 @@ _DIFFUSION_PRE_PROCESS_FUNCS = {
     "WanPipeline": "get_wan22_pre_process_func",
     "WanVACEPipeline": "get_wan22_vace_pre_process_func",
     "WanImageToVideoPipeline": "get_wan22_i2v_pre_process_func",
+    "WanS2VPipeline": "get_wan22_s2v_pre_process_func",
     "WanT2VDMD2Pipeline": "get_wan22_pre_process_func",
     "WanI2VDMD2Pipeline": "get_wan22_i2v_pre_process_func",
     "OmniGen2Pipeline": "get_omnigen2_pre_process_func",

vllm_omni/outputs.py

@@ -307,6 +307,72 @@ class OmniRequestOutput:
         """Check if this is a pipeline stage output."""
         return self.stage_id is not None and self.request_output is not None
 
+    def unwrap(self) -> "OmniRequestOutput":
+        """Unwrap nested OmniRequestOutput to get the innermost result.
+
+        This helper handles the common pattern where pipeline outputs may wrap
+        other OmniRequestOutput instances. It recursively unwraps until it reaches
+        the final output with actual content (images, text, etc.).
+
+        Returns:
+            The innermost OmniRequestOutput containing the actual generation results.
+
+        Example:
+            ```python
+            result = omni.generate(...)
+            output = OmniRequestOutput.unwrap_result(result)
+            if output.images:
+                # Access images directly
+                video_frames = output.images
+            ```
+        """
+        current = self
+        # Unwrap nested pipeline outputs
+        while current.is_pipeline_output and current.request_output is not None:
+            if isinstance(current.request_output, OmniRequestOutput):
+                current = current.request_output
+            else:
+                break
+        return current
+
+    @staticmethod
+    def unwrap_result(result: Any) -> "OmniRequestOutput":
+        """Unwrap result from omni.generate() to get the final OmniRequestOutput.
+
+        This static helper handles the full unwrapping pattern including:
+        1. Extracting from list if needed
+        2. Type validation
+        3. Recursive unwrapping of nested pipeline outputs
+
+        Args:
+            result: The result from omni.generate() - may be a list or OmniRequestOutput
+
+        Returns:
+            The innermost OmniRequestOutput with actual content
+
+        Raises:
+            ValueError: If result is not an OmniRequestOutput or list containing one
+
+        Example:
+            ```python
+            result = omni.generate(...)
+            output = OmniRequestOutput.unwrap_result(result)
+            # output is guaranteed to be the final OmniRequestOutput
+            ```
+        """
+        # Handle list wrapper
+        if isinstance(result, list):
+            if not result:
+                raise ValueError("Result list is empty")
+            result = result[0]
+
+        # Validate type
+        if not isinstance(result, OmniRequestOutput):
+            raise ValueError(f"Expected OmniRequestOutput, got {type(result)}")
+
+        # Unwrap nested outputs
+        return result.unwrap()
+
     def to_dict(self) -> dict[str, Any]:
         """Convert to dictionary for JSON serialization."""
         result = {

vllm_omni/platforms/xpu/platform.py

@@ -74,3 +74,8 @@ class XPUOmniPlatform(OmniPlatform, XPUPlatform):
     def get_free_memory(cls, device: torch.device | None = None) -> int:
         free, _ = torch.xpu.mem_get_info(device)
         return free
+
+    @classmethod
+    def get_profiler_cls(cls) -> str:
+        """Return XPU-specific profiler that handles XPU events."""
+        return "vllm_omni.platforms.xpu.profiler.XPUTorchProfilerWrapper"

vllm_omni/platforms/xpu/profiler.py

@@ -0,0 +1,109 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+from __future__ import annotations
+
+from typing_extensions import override
+from vllm.logger import init_logger
+
+from vllm_omni.profiler.omni_torch_profiler import (
+    OmniTorchProfilerWrapper,
+    TorchProfilerActivity,
+)
+
+logger = init_logger(__name__)
+
+
+class XPUTorchProfilerWrapper(OmniTorchProfilerWrapper):
+    """XPU-specific profiler wrapper.
+
+    Uses torch.profiler with XPU ProfilerActivity to capture Intel GPU events.
+    XPU events are exposed via torch.profiler.ProfilerActivity.XPU and work
+    with the standard torch.profiler interface.
+    """
+
+    @override
+    def _get_default_activities(self) -> list[TorchProfilerActivity]:
+        """Default to CPU + XPU profiling for Intel GPUs."""
+        return ["CPU", "XPU"]
+
+    @override
+    def _on_stop_hook(self) -> None:
+        """XPU-specific stop hook that handles XPU event aggregation.
+
+        XPU profiler follows CUDA-like patterns but uses xpu_time_total metrics.
+        """
+        rank = self.local_rank
+        sort_key = "self_xpu_time_total"
+
+        excel_sheets: dict[str, list[dict]] = {}
+
+        # 1) Summary op table
+        summary_events = self.profiler.key_averages()
+        excel_sheets["summary"] = self._event_list_to_rows(summary_events)
+
+        # 2) Shape-grouped op table
+        if self.profiler_config.torch_profiler_record_shapes:
+            try:
+                shape_events = self.profiler.key_averages(
+                    group_by_input_shape=True,
+                )
+                excel_sheets["by_shape"] = self._event_list_to_rows(shape_events)
+            except Exception as e:
+                logger.warning(
+                    "[Rank %s] Failed to export shape-grouped op table: %s",
+                    rank,
+                    e,
+                )
+
+        # 3) Stack-grouped op table
+        if self.profiler_config.torch_profiler_with_stack:
+            try:
+                stack_events = self.profiler.key_averages(
+                    group_by_stack_n=8,
+                )
+                excel_sheets["by_stack"] = self._event_list_to_rows(stack_events)
+            except Exception as e:
+                logger.warning(
+                    "[Rank %s] Failed to export stack-grouped op table: %s",
+                    rank,
+                    e,
+                )
+
+            # 4) Export stack files
+            try:
+                cpu_stack_file = self._artifact_path("stacks_cpu", ".txt")
+                self.profiler.export_stacks(
+                    cpu_stack_file,
+                    metric="self_cpu_time_total",
+                )
+                self._artifact_paths["stacks_cpu"] = cpu_stack_file
+            except Exception as e:
+                logger.warning("[Rank %s] export_stacks(cpu) failed: %s", rank, e)
+
+            # XPU stack traces
+            try:
+                xpu_stack_file = self._artifact_path("stacks_xpu", ".txt")
+                self.profiler.export_stacks(
+                    xpu_stack_file,
+                    metric="self_xpu_time_total",
+                )
+                self._artifact_paths["stacks_xpu"] = xpu_stack_file
+            except Exception as e:
+                logger.warning("[Rank %s] export_stacks(xpu) failed: %s", rank, e)
+
+        try:
+            self._table_path = self._write_excel_artifact("ops", excel_sheets)
+        except Exception as e:
+            logger.warning("[Rank %s] Failed to export Excel workbook: %s", rank, e)
+
+        # Print XPU time table for rank 0
+        if rank == 0:
+            try:
+                table = self.profiler.key_averages().table(sort_by=sort_key, row_limit=50)
+                logger.info("XPU Profiler Results:\n%s", table)
+            except Exception as e:
+                logger.warning("[Rank %s] Failed to generate XPU time table: %s", rank, e)
+
+        if self.dump_cpu_time_total and rank == 0:
+            logger.info(self.profiler.key_averages().table(sort_by="self_cpu_time_total", row_limit=50))