[Feature]Migrate helios example (#4569)
Signed-off-by: princepride <wangzhipeng628@gmail.com>
This commit is contained in:
18 files changed
+328
-1394
No files matched your search
@@ -18,7 +18,6 @@ nav:
|
||||
- Offline Inference:
|
||||
- BAGEL-7B-MoT: user_guide/examples/offline_inference/bagel.md
|
||||
- GLM-Image Multistage End-to-End Inference: user_guide/examples/offline_inference/glm_image.md
|
||||
- Helios Video Generation: user_guide/examples/offline_inference/helios.md
|
||||
- HunyuanImage-3.0 Image-to-Text Inference: user_guide/examples/offline_inference/hunyuan_image3.md
|
||||
- Image-To-Image: user_guide/examples/offline_inference/image_to_image.md
|
||||
- Image-To-Video: user_guide/examples/offline_inference/image_to_video.md
|
||||
|
||||
@@ -1,203 +0,0 @@
|
||||
# Helios Video Generation
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/helios>.
|
||||
|
||||
|
||||
Helios is a text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) diffusion model. This example demonstrates end-to-end video generation using vLLM-Omni with three model variants:
|
||||
|
||||
| Variant | Description | Key Features |
|
||||
|---------|-------------|--------------|
|
||||
| **Helios-Base** | Base model, Stage 1 only | Single-stage denoising, `guidance_scale=5.0` |
|
||||
| **Helios-Mid** | Mid model, Stage 2 pyramid | Multi-stage pyramid denoising, CFG-Zero* support |
|
||||
| **Helios-Distilled** | Distilled model, Stage 2+3 | Few-step inference with DMD, `guidance_scale=1.0` |
|
||||
|
||||
## Setup
|
||||
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
## Run Examples
|
||||
|
||||
Get into the example folder:
|
||||
```bash
|
||||
cd examples/offline_inference/helios
|
||||
```
|
||||
|
||||
### Text-to-Video (T2V)
|
||||
|
||||
**Helios-Base** (Stage 1 only):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_t2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 + CFG-Zero*):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1 \
|
||||
--output helios_t2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled** (Stage 2 pyramid + DMD):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk \
|
||||
--output helios_t2v_distilled.mp4
|
||||
```
|
||||
|
||||
### Image-to-Video (I2V)
|
||||
|
||||
**Helios-Base**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "A towering emerald wave surges forward, its crest curling with raw power and energy." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_i2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 + CFG-Zero*):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "A towering emerald wave surges forward, its crest curling with raw power and energy." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1 \
|
||||
--output helios_i2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "A towering emerald wave surges forward, its crest curling with raw power and energy." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk \
|
||||
--output helios_i2v_distilled.mp4
|
||||
```
|
||||
|
||||
### Video-to-Video (V2V)
|
||||
|
||||
**Helios-Base**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "A bright yellow Lamborghini speeds along a curving mountain road." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_v2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 + CFG-Zero*):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "A bright yellow Lamborghini speeds along a curving mountain road." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1 \
|
||||
--output helios_v2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "A bright yellow Lamborghini speeds along a curving mountain road." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk \
|
||||
--output helios_v2v_distilled.mp4
|
||||
```
|
||||
|
||||
## Common Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--model` | `BestWishYsh/Helios-Base` | Model ID or local path |
|
||||
| `--sample-type` | `t2v` | Generation mode: `t2v`, `i2v`, or `v2v` |
|
||||
| `--prompt` | — | Text prompt describing the video |
|
||||
| `--negative-prompt` | *(see source)* | Negative prompt for CFG (includes anti-static terms) |
|
||||
| `--image-path` | — | Input image (required for `i2v`) |
|
||||
| `--video-path` | — | Input video (required for `v2v`) |
|
||||
| `--height` | `384` | Video height in pixels |
|
||||
| `--width` | `640` | Video width in pixels |
|
||||
| `--num-frames` | `99` | Number of output frames |
|
||||
| `--num-inference-steps` | `50` | Denoising steps (Stage 1 only) |
|
||||
| `--guidance-scale` | `5.0` | CFG scale (`1.0` for Distilled) |
|
||||
| `--seed` | `42` | Random seed |
|
||||
| `--fps` | `16` | Output video frame rate |
|
||||
| `--output` | `helios_output.mp4` | Output file path |
|
||||
|
||||
### Stage 2 / Pyramid Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--is-enable-stage2` | off | Enable pyramid multi-stage denoising |
|
||||
| `--pyramid-num-stages` | `3` | Number of pyramid stages |
|
||||
| `--pyramid-num-inference-steps-list` | `10 10 10` | Steps per pyramid stage |
|
||||
| `--is-amplify-first-chunk` | off | DMD amplification (Distilled) |
|
||||
|
||||
### CFG-Zero* Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--use-cfg-zero-star` | off | Enable CFG-Zero* guidance (Mid) |
|
||||
| `--use-zero-init` | off | Zero init for first steps |
|
||||
| `--zero-steps` | `1` | Number of zero-init steps |
|
||||
|
||||
### Memory & Parallelism
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--vae-use-slicing` | off | Enable VAE slicing |
|
||||
| `--vae-use-tiling` | off | Enable VAE tiling |
|
||||
| `--enforce-eager` | off | Disable torch.compile |
|
||||
| `--enable-cpu-offload` | off | CPU offloading |
|
||||
| `--enable-layerwise-offload` | off | Layerwise offloading |
|
||||
| `--ulysses-degree` | `1` | Ulysses SP degree |
|
||||
| `--ring-degree` | `1` | Ring SP degree |
|
||||
| `--cfg-parallel-size` | `1` | CFG parallel size (1 or 2) |
|
||||
| `--tensor-parallel-size` | `1` | Tensor parallelism size |
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "end2end.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/helios/end2end.py"
|
||||
``````
|
||||
@@ -1,193 +0,0 @@
|
||||
# Helios Video Generation
|
||||
|
||||
Helios is a text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) diffusion model. This example demonstrates end-to-end video generation using vLLM-Omni with three model variants:
|
||||
|
||||
| Variant | Description | Key Features |
|
||||
|---------|-------------|--------------|
|
||||
| **Helios-Base** | Base model, Stage 1 only | Single-stage denoising, `guidance_scale=5.0` |
|
||||
| **Helios-Mid** | Mid model, Stage 2 pyramid | Multi-stage pyramid denoising, CFG-Zero* support |
|
||||
| **Helios-Distilled** | Distilled model, Stage 2+3 | Few-step inference with DMD, `guidance_scale=1.0` |
|
||||
|
||||
## Setup
|
||||
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
## Run Examples
|
||||
|
||||
Get into the example folder:
|
||||
```bash
|
||||
cd examples/offline_inference/helios
|
||||
```
|
||||
|
||||
### Text-to-Video (T2V)
|
||||
|
||||
**Helios-Base** (Stage 1 only):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_t2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 + CFG-Zero*):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1 \
|
||||
--output helios_t2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled** (Stage 2 pyramid + DMD):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk \
|
||||
--output helios_t2v_distilled.mp4
|
||||
```
|
||||
|
||||
### Image-to-Video (I2V)
|
||||
|
||||
**Helios-Base**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "A towering emerald wave surges forward, its crest curling with raw power and energy." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_i2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 + CFG-Zero*):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "A towering emerald wave surges forward, its crest curling with raw power and energy." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1 \
|
||||
--output helios_i2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "A towering emerald wave surges forward, its crest curling with raw power and energy." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk \
|
||||
--output helios_i2v_distilled.mp4
|
||||
```
|
||||
|
||||
### Video-to-Video (V2V)
|
||||
|
||||
**Helios-Base**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "A bright yellow Lamborghini speeds along a curving mountain road." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_v2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 + CFG-Zero*):
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "A bright yellow Lamborghini speeds along a curving mountain road." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1 \
|
||||
--output helios_v2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled**:
|
||||
```bash
|
||||
python end2end.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "A bright yellow Lamborghini speeds along a curving mountain road." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk \
|
||||
--output helios_v2v_distilled.mp4
|
||||
```
|
||||
|
||||
## Common Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--model` | `BestWishYsh/Helios-Base` | Model ID or local path |
|
||||
| `--sample-type` | `t2v` | Generation mode: `t2v`, `i2v`, or `v2v` |
|
||||
| `--prompt` | — | Text prompt describing the video |
|
||||
| `--negative-prompt` | *(see source)* | Negative prompt for CFG (includes anti-static terms) |
|
||||
| `--image-path` | — | Input image (required for `i2v`) |
|
||||
| `--video-path` | — | Input video (required for `v2v`) |
|
||||
| `--height` | `384` | Video height in pixels |
|
||||
| `--width` | `640` | Video width in pixels |
|
||||
| `--num-frames` | `99` | Number of output frames |
|
||||
| `--num-inference-steps` | `50` | Denoising steps (Stage 1 only) |
|
||||
| `--guidance-scale` | `5.0` | CFG scale (`1.0` for Distilled) |
|
||||
| `--seed` | `42` | Random seed |
|
||||
| `--fps` | `16` | Output video frame rate |
|
||||
| `--output` | `helios_output.mp4` | Output file path |
|
||||
|
||||
### Stage 2 / Pyramid Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--is-enable-stage2` | off | Enable pyramid multi-stage denoising |
|
||||
| `--pyramid-num-stages` | `3` | Number of pyramid stages |
|
||||
| `--pyramid-num-inference-steps-list` | `10 10 10` | Steps per pyramid stage |
|
||||
| `--is-amplify-first-chunk` | off | DMD amplification (Distilled) |
|
||||
|
||||
### CFG-Zero* Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--use-cfg-zero-star` | off | Enable CFG-Zero* guidance (Mid) |
|
||||
| `--use-zero-init` | off | Zero init for first steps |
|
||||
| `--zero-steps` | `1` | Number of zero-init steps |
|
||||
|
||||
### Memory & Parallelism
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--vae-use-slicing` | off | Enable VAE slicing |
|
||||
| `--vae-use-tiling` | off | Enable VAE tiling |
|
||||
| `--enforce-eager` | off | Disable torch.compile |
|
||||
| `--enable-cpu-offload` | off | CPU offloading |
|
||||
| `--enable-layerwise-offload` | off | Layerwise offloading |
|
||||
| `--ulysses-degree` | `1` | Ulysses SP degree |
|
||||
| `--ring-degree` | `1` | Ring SP degree |
|
||||
| `--cfg-parallel-size` | `1` | CFG parallel size (1 or 2) |
|
||||
| `--tensor-parallel-size` | `1` | Tensor parallelism size |
|
||||
@@ -1,355 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""
|
||||
Helios video generation example supporting T2V, I2V, and V2V.
|
||||
|
||||
Usage (T2V, Helios-Base, Stage 1 only):
|
||||
python end2end.py \
|
||||
--model /path/to/Helios-Base --sample-type t2v \
|
||||
--prompt "A serene lakeside sunrise with mist over the water." \
|
||||
--height 384 --width 640 --num-frames 99 \
|
||||
--num-inference-steps 50 --guidance-scale 5.0
|
||||
|
||||
Usage (I2V, Helios-Base):
|
||||
python end2end.py \
|
||||
--model /path/to/Helios-Base --sample-type i2v \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompt "Description of desired animation." \
|
||||
--guidance-scale 5.0
|
||||
|
||||
Usage (V2V, Helios-Base):
|
||||
python end2end.py \
|
||||
--model /path/to/Helios-Base --sample-type v2v \
|
||||
--video-path /path/to/video.mp4 \
|
||||
--prompt "Description of desired transformation." \
|
||||
--guidance-scale 5.0
|
||||
|
||||
Usage (Helios-Mid, Stage 2 + CFG-Zero*):
|
||||
python end2end.py \
|
||||
--model /path/to/Helios-Mid --sample-type t2v \
|
||||
--prompt "A serene lakeside sunrise with mist over the water." \
|
||||
--guidance-scale 5.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 20 20 20 \
|
||||
--use-cfg-zero-star --use-zero-init --zero-steps 1
|
||||
|
||||
Usage (Helios-Distilled, Stage 2 pyramid + DMD):
|
||||
python end2end.py \
|
||||
--model /path/to/Helios-Distilled --sample-type t2v \
|
||||
--prompt "A serene lakeside sunrise with mist over the water." \
|
||||
--guidance-scale 1.0 \
|
||||
--is-enable-stage2 \
|
||||
--pyramid-num-inference-steps-list 2 2 2 \
|
||||
--is-amplify-first-chunk
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.outputs import OmniRequestOutput
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
|
||||
def load_image_as_tensor(image_path: str, height: int, width: int) -> torch.Tensor:
|
||||
"""Load an image and return a (1, C, H, W) tensor normalized to [-1, 1]."""
|
||||
from PIL import Image
|
||||
from torchvision import transforms
|
||||
|
||||
image = Image.open(image_path).convert("RGB").resize((width, height))
|
||||
transform = transforms.Compose(
|
||||
[
|
||||
transforms.ToTensor(),
|
||||
transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
|
||||
]
|
||||
)
|
||||
return transform(image).unsqueeze(0)
|
||||
|
||||
|
||||
def load_video_as_tensor(video_path: str, height: int, width: int) -> torch.Tensor:
|
||||
"""Load a video and return a (1, C, T, H, W) tensor normalized to [-1, 1]."""
|
||||
from torchvision import transforms
|
||||
from torchvision.io import read_video
|
||||
|
||||
video_frames, _, _ = read_video(video_path, output_format="TCHW")
|
||||
video_frames = video_frames.float() / 255.0
|
||||
|
||||
transform = transforms.Compose(
|
||||
[
|
||||
transforms.Resize((height, width), antialias=True),
|
||||
transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
|
||||
]
|
||||
)
|
||||
video_frames = torch.stack([transform(f) for f in video_frames])
|
||||
return video_frames.permute(1, 0, 2, 3).unsqueeze(0)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Generate a video with Helios (T2V / I2V / V2V).")
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
default="BestWishYsh/Helios-Base",
|
||||
help="Helios model ID or local path (e.g. Helios-Base, Helios-Mid, Helios-Distilled).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sample-type",
|
||||
choices=["t2v", "i2v", "v2v"],
|
||||
default="t2v",
|
||||
help="Generation mode: t2v (text-to-video), i2v (image-to-video), v2v (video-to-video).",
|
||||
)
|
||||
parser.add_argument("--prompt", default="A serene lakeside sunrise with mist over the water.", help="Text prompt.")
|
||||
parser.add_argument(
|
||||
"--negative-prompt",
|
||||
default=(
|
||||
"Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, "
|
||||
"images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, "
|
||||
"incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, "
|
||||
"misshapen limbs, fused fingers, still picture, messy background, three legs, many people "
|
||||
"in the background, walking backwards"
|
||||
),
|
||||
help="Negative prompt.",
|
||||
)
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
|
||||
parser.add_argument("--guidance-scale", type=float, default=5.0, help="CFG scale.")
|
||||
parser.add_argument("--height", type=int, default=384, help="Video height.")
|
||||
parser.add_argument("--width", type=int, default=640, help="Video width.")
|
||||
parser.add_argument("--num-frames", type=int, default=99, help="Number of video frames.")
|
||||
parser.add_argument("--num-inference-steps", type=int, default=50, help="Sampling steps (Stage 1 only).")
|
||||
parser.add_argument("--output", type=str, default="helios_output.mp4", help="Output video path.")
|
||||
parser.add_argument("--fps", type=int, default=16, help="Frames per second for the output video.")
|
||||
|
||||
# I2V / V2V
|
||||
parser.add_argument("--image-path", type=str, default=None, help="Input image path for I2V mode.")
|
||||
parser.add_argument("--video-path", type=str, default=None, help="Input video path for V2V mode.")
|
||||
|
||||
# Stage 2 (pyramid multi-stage denoising)
|
||||
parser.add_argument(
|
||||
"--is-enable-stage2",
|
||||
action="store_true",
|
||||
help="Enable pyramid multi-stage denoising (Stage 2). Required for Helios-Distilled.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pyramid-num-stages",
|
||||
type=int,
|
||||
default=3,
|
||||
help="Number of pyramid stages for Stage 2.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pyramid-num-inference-steps-list",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=[10, 10, 10],
|
||||
help="Inference steps per pyramid stage.",
|
||||
)
|
||||
|
||||
# DMD
|
||||
parser.add_argument(
|
||||
"--is-amplify-first-chunk",
|
||||
action="store_true",
|
||||
help="Enable DMD amplification for the first chunk (Helios-Distilled).",
|
||||
)
|
||||
|
||||
# CFG Zero Star
|
||||
parser.add_argument(
|
||||
"--use-cfg-zero-star",
|
||||
action="store_true",
|
||||
help="Enable CFG Zero Star guidance (recommended for Helios-Mid).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-zero-init",
|
||||
action="store_true",
|
||||
help="Use zero initialization for the first denoising steps with CFG-Zero*.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--zero-steps",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of initial denoising steps using zero prediction (default: 1).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-diffusion-pipeline-profiler",
|
||||
action="store_true",
|
||||
help="Enable diffusion pipeline profiler to display stage durations.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--quantization",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Quantization method (e.g. 'fp8').",
|
||||
)
|
||||
|
||||
# Memory & parallelism
|
||||
parser.add_argument("--vae-use-slicing", action="store_true", help="Enable VAE slicing.")
|
||||
parser.add_argument("--vae-use-tiling", action="store_true", help="Enable VAE tiling.")
|
||||
parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
|
||||
parser.add_argument("--enable-cpu-offload", action="store_true", help="Enable CPU offloading.")
|
||||
parser.add_argument("--enable-layerwise-offload", action="store_true", help="Enable layerwise offloading.")
|
||||
parser.add_argument("--ulysses-degree", type=int, default=1, help="Ulysses SP degree.")
|
||||
parser.add_argument("--ring-degree", type=int, default=1, help="Ring SP degree.")
|
||||
parser.add_argument("--cfg-parallel-size", type=int, default=1, choices=[1, 2], help="CFG parallel size.")
|
||||
parser.add_argument("--tensor-parallel-size", type=int, default=1, help="Tensor parallelism size.")
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
|
||||
|
||||
parallel_config = DiffusionParallelConfig(
|
||||
ulysses_degree=args.ulysses_degree,
|
||||
ring_degree=args.ring_degree,
|
||||
cfg_parallel_size=args.cfg_parallel_size,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
)
|
||||
|
||||
omni = Omni(
|
||||
model=args.model,
|
||||
enable_layerwise_offload=args.enable_layerwise_offload,
|
||||
vae_use_slicing=args.vae_use_slicing,
|
||||
vae_use_tiling=args.vae_use_tiling,
|
||||
enable_cpu_offload=args.enable_cpu_offload,
|
||||
parallel_config=parallel_config,
|
||||
enforce_eager=args.enforce_eager,
|
||||
enable_diffusion_pipeline_profiler=args.enable_diffusion_pipeline_profiler,
|
||||
quantization=args.quantization,
|
||||
)
|
||||
|
||||
# Validate I2V / V2V arguments
|
||||
if args.sample_type == "i2v" and not args.image_path:
|
||||
raise ValueError("--image-path is required for I2V mode.")
|
||||
if args.sample_type == "v2v" and not args.video_path:
|
||||
raise ValueError("--video-path is required for V2V mode.")
|
||||
|
||||
# Build extra_args for Helios-specific parameters
|
||||
extra_args = {}
|
||||
if args.is_enable_stage2:
|
||||
extra_args["is_enable_stage2"] = True
|
||||
extra_args["pyramid_num_stages"] = args.pyramid_num_stages
|
||||
extra_args["pyramid_num_inference_steps_list"] = args.pyramid_num_inference_steps_list
|
||||
if args.is_amplify_first_chunk:
|
||||
extra_args["is_amplify_first_chunk"] = True
|
||||
if args.use_cfg_zero_star:
|
||||
extra_args["use_cfg_zero_star"] = True
|
||||
if args.use_zero_init:
|
||||
extra_args["use_zero_init"] = True
|
||||
extra_args["zero_steps"] = args.zero_steps
|
||||
|
||||
if args.sample_type == "i2v":
|
||||
image_tensor = load_image_as_tensor(args.image_path, args.height, args.width)
|
||||
extra_args["image"] = image_tensor
|
||||
elif args.sample_type == "v2v":
|
||||
video_tensor = load_video_as_tensor(args.video_path, args.height, args.width)
|
||||
extra_args["video"] = video_tensor
|
||||
|
||||
# Print generation configuration
|
||||
print(f"\n{'=' * 60}")
|
||||
print("Helios Generation Configuration:")
|
||||
print(f" Model: {args.model}")
|
||||
print(f" Sample type: {args.sample_type.upper()}")
|
||||
if args.sample_type == "i2v":
|
||||
print(f" Image: {args.image_path}")
|
||||
elif args.sample_type == "v2v":
|
||||
print(f" Video: {args.video_path}")
|
||||
print(f" Prompt: {args.prompt[:80]}{'...' if len(args.prompt) > 80 else ''}")
|
||||
print(f" Video size: {args.width}x{args.height}, {args.num_frames} frames")
|
||||
print(f" Inference steps: {args.num_inference_steps}")
|
||||
print(f" Guidance scale: {args.guidance_scale}")
|
||||
if args.is_enable_stage2:
|
||||
print(f" Stage 2: enabled (stages={args.pyramid_num_stages}, steps={args.pyramid_num_inference_steps_list})")
|
||||
if args.is_amplify_first_chunk:
|
||||
print(" DMD amplify first chunk: enabled")
|
||||
if args.use_cfg_zero_star:
|
||||
print(f" CFG Zero Star: enabled (zero_init={args.use_zero_init}, zero_steps={args.zero_steps})")
|
||||
else:
|
||||
if args.use_cfg_zero_star:
|
||||
print(f" CFG Zero Star: enabled (zero_init={args.use_zero_init}, zero_steps={args.zero_steps})")
|
||||
print(" Stage 2: disabled (Stage 1 only)")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
generation_start = time.perf_counter()
|
||||
frames = omni.generate(
|
||||
{
|
||||
"prompt": args.prompt,
|
||||
"negative_prompt": args.negative_prompt,
|
||||
},
|
||||
OmniDiffusionSamplingParams(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
generator=generator,
|
||||
guidance_scale=args.guidance_scale,
|
||||
num_inference_steps=args.num_inference_steps,
|
||||
num_frames=args.num_frames,
|
||||
extra_args=extra_args,
|
||||
),
|
||||
)
|
||||
generation_end = time.perf_counter()
|
||||
generation_time = generation_end - generation_start
|
||||
|
||||
print(f"Total generation time: {generation_time:.4f} seconds ({generation_time * 1000:.2f} ms)")
|
||||
|
||||
# Extract video frames from OmniRequestOutput
|
||||
if isinstance(frames, list) and len(frames) > 0:
|
||||
first_item = frames[0]
|
||||
|
||||
if hasattr(first_item, "final_output_type"):
|
||||
if first_item.final_output_type != "image":
|
||||
raise ValueError(
|
||||
f"Unexpected output type '{first_item.final_output_type}', expected 'image' for video generation."
|
||||
)
|
||||
|
||||
if hasattr(first_item, "is_pipeline_output") and first_item.is_pipeline_output:
|
||||
inner_output = first_item.request_output
|
||||
if isinstance(inner_output, OmniRequestOutput) and hasattr(inner_output, "images"):
|
||||
frames = inner_output.images[0] if inner_output.images else None
|
||||
if frames is None:
|
||||
raise ValueError("No video frames found in output.")
|
||||
elif hasattr(first_item, "images") and first_item.images:
|
||||
frames = first_item.images
|
||||
else:
|
||||
raise ValueError("No video frames found in OmniRequestOutput.")
|
||||
|
||||
# Unwrap batch list from postprocess_video: [numpy(T,H,W,C)] -> numpy(T,H,W,C)
|
||||
if isinstance(frames, list) and len(frames) > 0 and isinstance(frames[0], np.ndarray):
|
||||
frames = frames[0]
|
||||
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
from diffusers.utils import export_to_video
|
||||
except ImportError:
|
||||
raise ImportError("diffusers is required for export_to_video.")
|
||||
|
||||
if isinstance(frames, torch.Tensor):
|
||||
video_tensor = frames.detach().cpu()
|
||||
if video_tensor.dim() == 5:
|
||||
if video_tensor.shape[1] in (3, 4):
|
||||
video_tensor = video_tensor[0].permute(1, 2, 3, 0)
|
||||
else:
|
||||
video_tensor = video_tensor[0]
|
||||
elif video_tensor.dim() == 4 and video_tensor.shape[0] in (3, 4):
|
||||
video_tensor = video_tensor.permute(1, 2, 3, 0)
|
||||
if video_tensor.is_floating_point():
|
||||
video_tensor = video_tensor.clamp(-1, 1) * 0.5 + 0.5
|
||||
video_array = video_tensor.float().numpy()
|
||||
elif isinstance(frames, np.ndarray):
|
||||
video_array = frames
|
||||
else:
|
||||
video_array = frames
|
||||
|
||||
if isinstance(video_array, np.ndarray) and video_array.ndim == 5:
|
||||
video_array = video_array[0]
|
||||
|
||||
export_to_video(video_array, str(output_path), fps=args.fps)
|
||||
print(f"Saved generated video to {output_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -9,6 +9,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
|
||||
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB |
|
||||
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
|
||||
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
|
||||
| `BestWishYsh/Helios-Base` / `Helios-Mid` / `Helios-Distilled` | 384x640 | 99 | 50 | 5.0 / 5.0 / 1.0 | — |
|
||||
|
||||
## Local CLI Usage
|
||||
|
||||
@@ -100,6 +101,49 @@ python text_to_video.py \
|
||||
--output quick_test.mp4
|
||||
```
|
||||
|
||||
### Helios (T2V)
|
||||
|
||||
Helios ships three variants. Model-specific knobs (declared in
|
||||
`vllm_omni/model_extras/helios.py`) are passed via the generic `--extra-body`
|
||||
JSON flag rather than bespoke per-model flags.
|
||||
|
||||
**Helios-Base** (Stage 1 only):
|
||||
|
||||
```bash
|
||||
python text_to_video.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--prompt "A dynamic time-lapse of scenery rushing past the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_t2v_base.mp4
|
||||
```
|
||||
|
||||
**Helios-Mid** (Stage 2 pyramid + CFG-Zero*):
|
||||
|
||||
```bash
|
||||
python text_to_video.py \
|
||||
--model BestWishYsh/Helios-Mid \
|
||||
--prompt "A dynamic time-lapse of scenery rushing past the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--extra-body '{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [20, 20, 20], "use_cfg_zero_star": true, "use_zero_init": true, "zero_steps": 1}' \
|
||||
--output helios_t2v_mid.mp4
|
||||
```
|
||||
|
||||
**Helios-Distilled** (Stage 2 pyramid + DMD, few-step):
|
||||
|
||||
```bash
|
||||
python text_to_video.py \
|
||||
--model BestWishYsh/Helios-Distilled \
|
||||
--prompt "A dynamic time-lapse of scenery rushing past the window of a speeding train." \
|
||||
--num-frames 240 \
|
||||
--guidance-scale 1.0 \
|
||||
--extra-body '{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [2, 2, 2], "is_amplify_first_chunk": true}' \
|
||||
--output helios_t2v_distilled.mp4
|
||||
```
|
||||
|
||||
> Helios image-to-video (I2V) and video-to-video (V2V) require image/video
|
||||
> conditioning tensors that cannot be passed through the JSON `--extra-body`
|
||||
> flag; they are out of scope for this text-to-video example.
|
||||
|
||||
## Key Arguments
|
||||
|
||||
### Common
|
||||
@@ -112,6 +156,7 @@ python text_to_video.py \
|
||||
- `--num-inference-steps`: sampling steps. Default depends on model.
|
||||
- `--fps`: frames per second for the saved MP4.
|
||||
- `--output`: path to save the generated video.
|
||||
- `--extra-body`: JSON dict of model-specific knobs (declared in `vllm_omni/model_extras/`), merged into sampling `extra_args`. See the Helios recipes above.
|
||||
- `--vae-use-slicing`: enable VAE slicing for memory optimization.
|
||||
- `--vae-use-tiling`: enable VAE tiling for memory optimization.
|
||||
- `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](../../../docs/user_guide/diffusion/parallelism_acceleration.md#cfg-parallel).
|
||||
|
||||
@@ -35,6 +35,15 @@ _MODEL_PRESETS = {
|
||||
"fps": 24,
|
||||
"output": "hunyuan_video_15_output.mp4",
|
||||
},
|
||||
"helios": {
|
||||
"height": 384,
|
||||
"width": 640,
|
||||
"num_frames": 99,
|
||||
"num_inference_steps": 50,
|
||||
"guidance_scale": 5.0,
|
||||
"fps": 16,
|
||||
"output": "helios_output.mp4",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -42,6 +51,8 @@ def _detect_preset(model: str) -> dict:
|
||||
model_lower = model.lower()
|
||||
if "hunyuan" in model_lower:
|
||||
return _MODEL_PRESETS["hunyuan"]
|
||||
if "helios" in model_lower:
|
||||
return _MODEL_PRESETS["helios"]
|
||||
return _MODEL_PRESETS["wan"]
|
||||
|
||||
|
||||
@@ -55,10 +66,26 @@ def parse_profiler_config(value: str) -> dict[str, Any]:
|
||||
return config
|
||||
|
||||
|
||||
def parse_extra_body(value: str) -> dict[str, Any]:
|
||||
"""Parse a JSON object of model-specific extra_body params.
|
||||
|
||||
Pipeline-declared knobs (see vllm_omni/model_extras/) are merged into
|
||||
OmniDiffusionSamplingParams.extra_args, so a single generic example can
|
||||
drive model-specific behaviour without bespoke per-model flags.
|
||||
"""
|
||||
try:
|
||||
body = json.loads(value)
|
||||
except json.JSONDecodeError as e:
|
||||
raise argparse.ArgumentTypeError(f"--extra-body must be valid JSON: {e}") from e
|
||||
if not isinstance(body, dict):
|
||||
raise argparse.ArgumentTypeError("--extra-body must be a JSON object")
|
||||
return body
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate a video from a text prompt. "
|
||||
"Supports Wan2.2, HunyuanVideo-1.5, and other text-to-video models."
|
||||
"Supports Wan2.2, HunyuanVideo-1.5, Helios, and other text-to-video models."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
@@ -74,6 +101,14 @@ def parse_args() -> argparse.Namespace:
|
||||
)
|
||||
parser.add_argument("--prompt", default="A serene lakeside sunrise with mist over the water.", help="Text prompt.")
|
||||
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
|
||||
parser.add_argument(
|
||||
"--extra-body",
|
||||
type=parse_extra_body,
|
||||
default=None,
|
||||
help="JSON dict of model-specific extra_body params (declared in vllm_omni/model_extras/), "
|
||||
"merged into sampling extra_args. Example (Helios-Distilled): "
|
||||
'\'{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [2, 2, 2], "is_amplify_first_chunk": true}\'.',
|
||||
)
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
|
||||
parser.add_argument("--guidance-scale", type=float, default=None, help="CFG scale. Default: model-specific.")
|
||||
parser.add_argument(
|
||||
@@ -357,6 +392,9 @@ def main():
|
||||
)
|
||||
if args.guidance_scale_high is not None:
|
||||
sampling_kwargs["guidance_scale_2"] = args.guidance_scale_high
|
||||
if args.extra_body:
|
||||
# Model-specific knobs (declared in vllm_omni/model_extras/) routed via extra_args.
|
||||
sampling_kwargs["extra_args"] = dict(args.extra_body)
|
||||
|
||||
generation_start = time.perf_counter()
|
||||
frames = omni.generate(
|
||||
|
||||
@@ -1,175 +0,0 @@
|
||||
# Helios Video Generation via Online API
|
||||
|
||||
This example demonstrates how to use the `/v1/videos` API with Helios models using the generic `extra_params` field.
|
||||
|
||||
## Overview
|
||||
|
||||
The `/v1/videos` API now supports model-specific parameters through the `extra_params` field, which accepts a JSON object containing any model-specific configuration. This allows supporting new models like Helios without modifying the API.
|
||||
|
||||
## Helios Model Variants
|
||||
|
||||
- **Helios-Base**: Basic T2V/I2V/V2V generation (Stage 1 only)
|
||||
- **Helios-Mid**: Advanced generation with CFG-Zero* (Stage 2)
|
||||
- **Helios-Distilled**: Fast generation with DMD (Stage 2)
|
||||
|
||||
## API Usage Examples
|
||||
|
||||
### 1. T2V (Text-to-Video) - Helios-Base
|
||||
|
||||
Basic text-to-video generation:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/videos \
|
||||
-F "prompt=A serene lakeside sunrise with mist over the water." \
|
||||
-F "model=BestWishYsh/Helios-Base" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "num_frames=99" \
|
||||
-F "num_inference_steps=50" \
|
||||
-F "guidance_scale=5.0" \
|
||||
-F "seed=42"
|
||||
```
|
||||
|
||||
### 2. I2V (Image-to-Video) - Helios-Base
|
||||
|
||||
Generate video from an input image:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/videos \
|
||||
-F "prompt=The lake water gently ripples as morning mist rises." \
|
||||
-F "model=BestWishYsh/Helios-Base" \
|
||||
-F "input_reference=@/path/to/image.jpg" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "num_frames=99" \
|
||||
-F "guidance_scale=5.0"
|
||||
```
|
||||
|
||||
### 3. Helios-Mid with Stage 2 + CFG-Zero*
|
||||
|
||||
Advanced generation with pyramid multi-stage denoising:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/videos \
|
||||
-F "prompt=A serene lakeside sunrise with mist over the water." \
|
||||
-F "model=BestWishYsh/Helios-Mid" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "guidance_scale=5.0" \
|
||||
-F 'extra_params={
|
||||
"is_enable_stage2": true,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [20, 20, 20],
|
||||
"use_cfg_zero_star": true,
|
||||
"use_zero_init": true,
|
||||
"zero_steps": 1
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. Helios-Distilled with DMD
|
||||
|
||||
Fast generation with Distribution Matching Distillation:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/videos \
|
||||
-F "prompt=A serene lakeside sunrise with mist over the water." \
|
||||
-F "model=BestWishYsh/Helios-Distilled" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "guidance_scale=1.0" \
|
||||
-F 'extra_params={
|
||||
"is_enable_stage2": true,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [2, 2, 2],
|
||||
"is_amplify_first_chunk": true
|
||||
}'
|
||||
```
|
||||
|
||||
## Model-Specific Parameters
|
||||
|
||||
The `extra_params` field accepts a JSON object with model-specific parameters. For Helios models, supported parameters include:
|
||||
|
||||
### Stage 2 Parameters
|
||||
- `is_enable_stage2` (bool): Enable pyramid multi-stage denoising
|
||||
- `pyramid_num_stages` (int): Number of pyramid stages (default: 3)
|
||||
- `pyramid_num_inference_steps_list` (array): Steps per stage, e.g., `[20, 20, 20]`
|
||||
|
||||
### CFG Zero Star Parameters (Helios-Mid)
|
||||
- `use_cfg_zero_star` (bool): Enable CFG Zero Star guidance
|
||||
- `use_zero_init` (bool): Use zero initialization for first steps
|
||||
- `zero_steps` (int): Number of initial zero prediction steps (default: 1)
|
||||
|
||||
### DMD Parameters (Helios-Distilled)
|
||||
- `is_amplify_first_chunk` (bool): Enable DMD amplification for first chunk
|
||||
|
||||
### Video Input (V2V mode)
|
||||
For video-to-video generation, upload a video file via `input_reference`:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/videos \
|
||||
-F "prompt=Transform the video into a watercolor painting style." \
|
||||
-F "model=BestWishYsh/Helios-Base" \
|
||||
-F "input_reference=@/path/to/video.mp4" \
|
||||
-F "guidance_scale=5.0"
|
||||
```
|
||||
|
||||
## Python Example
|
||||
|
||||
```python
|
||||
import requests
|
||||
import json
|
||||
|
||||
url = "http://localhost:8000/v1/videos"
|
||||
|
||||
# Helios-Mid with Stage 2
|
||||
data = {
|
||||
"prompt": "A serene lakeside sunrise with mist over the water.",
|
||||
"model": "BestWishYsh/Helios-Mid",
|
||||
"width": 640,
|
||||
"height": 384,
|
||||
"guidance_scale": 5.0,
|
||||
"extra_params": json.dumps({
|
||||
"is_enable_stage2": True,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [20, 20, 20],
|
||||
"use_cfg_zero_star": True,
|
||||
"use_zero_init": True,
|
||||
"zero_steps": 1
|
||||
})
|
||||
}
|
||||
|
||||
response = requests.post(url, data=data)
|
||||
video_job = response.json()
|
||||
print(f"Video job created: {video_job['id']}")
|
||||
|
||||
# Poll for completion
|
||||
import time
|
||||
while True:
|
||||
status_response = requests.get(f"{url}/{video_job['id']}")
|
||||
status = status_response.json()
|
||||
|
||||
if status['status'] == 'completed':
|
||||
print(f"Video generated: {status['file_name']}")
|
||||
break
|
||||
elif status['status'] == 'failed':
|
||||
print(f"Generation failed: {status.get('error')}")
|
||||
break
|
||||
|
||||
print(f"Progress: {status['progress']}%")
|
||||
time.sleep(2)
|
||||
```
|
||||
|
||||
## Benefits of This Approach
|
||||
|
||||
1. **No API Changes**: New models can be supported without modifying the API
|
||||
2. **Backward Compatible**: Existing clients continue to work
|
||||
3. **Flexible**: Any model-specific parameter can be passed
|
||||
4. **Type-Safe**: Parameters are validated at the model level
|
||||
5. **Future-Proof**: Supports models that don't exist yet
|
||||
|
||||
## Notes
|
||||
|
||||
- The `extra_params` field must be a valid JSON object
|
||||
- Parameters are passed directly to the model's `extra_args`
|
||||
- Invalid parameters will be caught by the model implementation
|
||||
- Tensor inputs (images/videos) should use `input_reference`, not `extra_params`
|
||||
@@ -1,212 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""
|
||||
Helios video generation client example using the /v1/videos API.
|
||||
|
||||
This example demonstrates how to use the generic extra_params field
|
||||
to pass Helios-specific parameters without modifying the API.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
|
||||
def create_video_job(
|
||||
api_url: str,
|
||||
prompt: str,
|
||||
model: str,
|
||||
width: int = 640,
|
||||
height: int = 384,
|
||||
num_frames: int = 99,
|
||||
guidance_scale: float = 5.0,
|
||||
seed: int = 42,
|
||||
extra_params: dict | None = None,
|
||||
input_image: Path | None = None,
|
||||
) -> dict:
|
||||
"""Create a video generation job."""
|
||||
data = {
|
||||
"prompt": prompt,
|
||||
"model": model,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"num_frames": num_frames,
|
||||
"guidance_scale": guidance_scale,
|
||||
"seed": seed,
|
||||
}
|
||||
|
||||
files = {}
|
||||
if input_image:
|
||||
files["input_reference"] = open(input_image, "rb")
|
||||
|
||||
if extra_params:
|
||||
data["extra_params"] = json.dumps(extra_params)
|
||||
|
||||
response = requests.post(f"{api_url}/v1/videos", data=data, files=files)
|
||||
response.raise_for_status()
|
||||
|
||||
if files:
|
||||
files["input_reference"].close()
|
||||
|
||||
return response.json()
|
||||
|
||||
|
||||
def poll_video_status(api_url: str, video_id: str, poll_interval: int = 2) -> dict:
|
||||
"""Poll video generation status until completion."""
|
||||
print(f"Polling video job: {video_id}")
|
||||
|
||||
while True:
|
||||
response = requests.get(f"{api_url}/v1/videos/{video_id}")
|
||||
response.raise_for_status()
|
||||
status_data = response.json()
|
||||
|
||||
status = status_data["status"]
|
||||
progress = status_data.get("progress", 0)
|
||||
|
||||
print(f"Status: {status}, Progress: {progress}%")
|
||||
|
||||
if status == "completed":
|
||||
print("Video generation completed!")
|
||||
return status_data
|
||||
elif status == "failed":
|
||||
error = status_data.get("error", {})
|
||||
raise RuntimeError(f"Video generation failed: {error}")
|
||||
|
||||
time.sleep(poll_interval)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Helios video generation client")
|
||||
parser.add_argument(
|
||||
"--api-url",
|
||||
default="http://localhost:8000",
|
||||
help="API server URL",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
default="BestWishYsh/Helios-Base",
|
||||
help="Model name (Helios-Base, Helios-Mid, Helios-Distilled)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
default="A serene lakeside sunrise with mist over the water.",
|
||||
help="Text prompt",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--width",
|
||||
type=int,
|
||||
default=640,
|
||||
help="Video width",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--height",
|
||||
type=int,
|
||||
default=384,
|
||||
help="Video height",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-frames",
|
||||
type=int,
|
||||
default=99,
|
||||
help="Number of frames",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--guidance-scale",
|
||||
type=float,
|
||||
default=5.0,
|
||||
help="Guidance scale",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--seed",
|
||||
type=int,
|
||||
default=42,
|
||||
help="Random seed",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input-image",
|
||||
type=Path,
|
||||
help="Input image for I2V mode",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--preset",
|
||||
choices=["base", "mid-stage2", "distilled"],
|
||||
default="base",
|
||||
help="Helios preset configuration",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Define preset configurations
|
||||
presets = {
|
||||
"base": None, # No extra params for base model
|
||||
"mid-stage2": {
|
||||
"is_enable_stage2": True,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [20, 20, 20],
|
||||
"use_cfg_zero_star": True,
|
||||
"use_zero_init": True,
|
||||
"zero_steps": 1,
|
||||
},
|
||||
"distilled": {
|
||||
"is_enable_stage2": True,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [2, 2, 2],
|
||||
"is_amplify_first_chunk": True,
|
||||
},
|
||||
}
|
||||
|
||||
extra_params = presets[args.preset]
|
||||
|
||||
print("=" * 50)
|
||||
print("Helios Video Generation")
|
||||
print("=" * 50)
|
||||
print(f"API URL: {args.api_url}")
|
||||
print(f"Model: {args.model}")
|
||||
print(f"Preset: {args.preset}")
|
||||
print(f"Prompt: {args.prompt}")
|
||||
print(f"Size: {args.width}x{args.height}")
|
||||
print(f"Frames: {args.num_frames}")
|
||||
print(f"Guidance Scale: {args.guidance_scale}")
|
||||
if extra_params:
|
||||
print(f"Extra Params: {json.dumps(extra_params, indent=2)}")
|
||||
print()
|
||||
|
||||
# Create video job
|
||||
print("Creating video generation job...")
|
||||
job = create_video_job(
|
||||
api_url=args.api_url,
|
||||
prompt=args.prompt,
|
||||
model=args.model,
|
||||
width=args.width,
|
||||
height=args.height,
|
||||
num_frames=args.num_frames,
|
||||
guidance_scale=args.guidance_scale,
|
||||
seed=args.seed,
|
||||
extra_params=extra_params,
|
||||
input_image=args.input_image,
|
||||
)
|
||||
|
||||
video_id = job["id"]
|
||||
print(f"Video job created: {video_id}")
|
||||
print()
|
||||
|
||||
# Poll for completion
|
||||
result = poll_video_status(args.api_url, video_id)
|
||||
|
||||
print()
|
||||
print("=" * 50)
|
||||
print("Generation Complete")
|
||||
print("=" * 50)
|
||||
print(f"Video ID: {result['id']}")
|
||||
print(f"File: {result.get('file_name', 'N/A')}")
|
||||
print(f"Inference Time: {result.get('inference_time_s', 'N/A')}s")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,79 +0,0 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
# Helios-Distilled with DMD Example
|
||||
# Fast generation with Distribution Matching Distillation
|
||||
|
||||
API_URL="${API_URL:-http://localhost:8000}"
|
||||
MODEL="${MODEL:-BestWishYsh/Helios-Distilled}"
|
||||
PROMPT="${PROMPT:-A serene lakeside sunrise with mist over the water.}"
|
||||
|
||||
echo "==================================="
|
||||
echo "Helios-Distilled with DMD"
|
||||
echo "==================================="
|
||||
echo "API URL: $API_URL"
|
||||
echo "Model: $MODEL"
|
||||
echo "Prompt: $PROMPT"
|
||||
echo ""
|
||||
|
||||
# Model-specific parameters for Helios-Distilled
|
||||
extra_params='{
|
||||
"is_enable_stage2": true,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [2, 2, 2],
|
||||
"is_amplify_first_chunk": true
|
||||
}'
|
||||
|
||||
echo "Model extra params: $extra_params"
|
||||
echo ""
|
||||
|
||||
# Create video generation job
|
||||
echo "Creating video generation job..."
|
||||
RESPONSE=$(curl -s -X POST "$API_URL/v1/videos" \
|
||||
-F "prompt=$PROMPT" \
|
||||
-F "model=$MODEL" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "guidance_scale=1.0" \
|
||||
-F "seed=42" \
|
||||
-F "extra_params=$extra_params")
|
||||
|
||||
echo "Response: $RESPONSE"
|
||||
echo ""
|
||||
|
||||
# Extract video ID
|
||||
VIDEO_ID=$(echo "$RESPONSE" | grep -o '"id":"[^"]*"' | cut -d'"' -f4)
|
||||
|
||||
if [ -z "$VIDEO_ID" ]; then
|
||||
echo "Error: Failed to create video job"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Video job created: $VIDEO_ID"
|
||||
echo ""
|
||||
|
||||
# Poll for completion
|
||||
echo "Polling for completion..."
|
||||
while true; do
|
||||
STATUS_RESPONSE=$(curl -s "$API_URL/v1/videos/$VIDEO_ID")
|
||||
STATUS=$(echo "$STATUS_RESPONSE" | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
|
||||
PROGRESS=$(echo "$STATUS_RESPONSE" | grep -o '"progress":[0-9]*' | cut -d':' -f2)
|
||||
|
||||
echo "Status: $STATUS, Progress: $PROGRESS%"
|
||||
|
||||
if [ "$STATUS" = "completed" ]; then
|
||||
echo ""
|
||||
echo "Video generation completed!"
|
||||
echo "Full response:"
|
||||
echo "$STATUS_RESPONSE" | jq '.'
|
||||
break
|
||||
elif [ "$STATUS" = "failed" ]; then
|
||||
echo ""
|
||||
echo "Video generation failed!"
|
||||
echo "$STATUS_RESPONSE" | jq '.'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
@@ -1,81 +0,0 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
# Helios-Mid with Stage 2 + CFG-Zero* Example
|
||||
# This demonstrates advanced generation with pyramid multi-stage denoising
|
||||
|
||||
API_URL="${API_URL:-http://localhost:8000}"
|
||||
MODEL="${MODEL:-BestWishYsh/Helios-Mid}"
|
||||
PROMPT="${PROMPT:-A serene lakeside sunrise with mist over the water.}"
|
||||
|
||||
echo "==================================="
|
||||
echo "Helios-Mid Stage 2 + CFG-Zero*"
|
||||
echo "==================================="
|
||||
echo "API URL: $API_URL"
|
||||
echo "Model: $MODEL"
|
||||
echo "Prompt: $PROMPT"
|
||||
echo ""
|
||||
|
||||
# Model-specific parameters for Helios-Mid
|
||||
extra_params='{
|
||||
"is_enable_stage2": true,
|
||||
"pyramid_num_stages": 3,
|
||||
"pyramid_num_inference_steps_list": [20, 20, 20],
|
||||
"use_cfg_zero_star": true,
|
||||
"use_zero_init": true,
|
||||
"zero_steps": 1
|
||||
}'
|
||||
|
||||
echo "Model extra params: $extra_params"
|
||||
echo ""
|
||||
|
||||
# Create video generation job
|
||||
echo "Creating video generation job..."
|
||||
RESPONSE=$(curl -s -X POST "$API_URL/v1/videos" \
|
||||
-F "prompt=$PROMPT" \
|
||||
-F "model=$MODEL" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "guidance_scale=5.0" \
|
||||
-F "seed=42" \
|
||||
-F "extra_params=$extra_params")
|
||||
|
||||
echo "Response: $RESPONSE"
|
||||
echo ""
|
||||
|
||||
# Extract video ID
|
||||
VIDEO_ID=$(echo "$RESPONSE" | grep -o '"id":"[^"]*"' | cut -d'"' -f4)
|
||||
|
||||
if [ -z "$VIDEO_ID" ]; then
|
||||
echo "Error: Failed to create video job"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Video job created: $VIDEO_ID"
|
||||
echo ""
|
||||
|
||||
# Poll for completion
|
||||
echo "Polling for completion..."
|
||||
while true; do
|
||||
STATUS_RESPONSE=$(curl -s "$API_URL/v1/videos/$VIDEO_ID")
|
||||
STATUS=$(echo "$STATUS_RESPONSE" | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
|
||||
PROGRESS=$(echo "$STATUS_RESPONSE" | grep -o '"progress":[0-9]*' | cut -d':' -f2)
|
||||
|
||||
echo "Status: $STATUS, Progress: $PROGRESS%"
|
||||
|
||||
if [ "$STATUS" = "completed" ]; then
|
||||
echo ""
|
||||
echo "Video generation completed!"
|
||||
echo "Full response:"
|
||||
echo "$STATUS_RESPONSE" | jq '.'
|
||||
break
|
||||
elif [ "$STATUS" = "failed" ]; then
|
||||
echo ""
|
||||
echo "Video generation failed!"
|
||||
echo "$STATUS_RESPONSE" | jq '.'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
@@ -1,69 +0,0 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
# Helios Text-to-Video Generation Example
|
||||
# This script demonstrates how to use the /v1/videos API with Helios models
|
||||
|
||||
API_URL="${API_URL:-http://localhost:8000}"
|
||||
MODEL="${MODEL:-BestWishYsh/Helios-Base}"
|
||||
PROMPT="${PROMPT:-A serene lakeside sunrise with mist over the water.}"
|
||||
|
||||
echo "==================================="
|
||||
echo "Helios T2V Generation"
|
||||
echo "==================================="
|
||||
echo "API URL: $API_URL"
|
||||
echo "Model: $MODEL"
|
||||
echo "Prompt: $PROMPT"
|
||||
echo ""
|
||||
|
||||
# Create video generation job
|
||||
echo "Creating video generation job..."
|
||||
RESPONSE=$(curl -s -X POST "$API_URL/v1/videos" \
|
||||
-F "prompt=$PROMPT" \
|
||||
-F "model=$MODEL" \
|
||||
-F "width=640" \
|
||||
-F "height=384" \
|
||||
-F "num_frames=99" \
|
||||
-F "num_inference_steps=50" \
|
||||
-F "guidance_scale=5.0" \
|
||||
-F "seed=42")
|
||||
|
||||
echo "Response: $RESPONSE"
|
||||
echo ""
|
||||
|
||||
# Extract video ID
|
||||
VIDEO_ID=$(echo "$RESPONSE" | grep -o '"id":"[^"]*"' | cut -d'"' -f4)
|
||||
|
||||
if [ -z "$VIDEO_ID" ]; then
|
||||
echo "Error: Failed to create video job"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Video job created: $VIDEO_ID"
|
||||
echo ""
|
||||
|
||||
# Poll for completion
|
||||
echo "Polling for completion..."
|
||||
while true; do
|
||||
STATUS_RESPONSE=$(curl -s "$API_URL/v1/videos/$VIDEO_ID")
|
||||
STATUS=$(echo "$STATUS_RESPONSE" | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
|
||||
PROGRESS=$(echo "$STATUS_RESPONSE" | grep -o '"progress":[0-9]*' | cut -d':' -f2)
|
||||
|
||||
echo "Status: $STATUS, Progress: $PROGRESS%"
|
||||
|
||||
if [ "$STATUS" = "completed" ]; then
|
||||
echo ""
|
||||
echo "Video generation completed!"
|
||||
echo "Full response:"
|
||||
echo "$STATUS_RESPONSE" | jq '.'
|
||||
break
|
||||
elif [ "$STATUS" = "failed" ]; then
|
||||
echo ""
|
||||
echo "Video generation failed!"
|
||||
echo "$STATUS_RESPONSE" | jq '.'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
@@ -10,6 +10,7 @@ This example demonstrates how to deploy text-to-video models for online video ge
|
||||
| Wan2.1 T2V (14B) | `Wan-AI/Wan2.1-T2V-14B-Diffusers` |
|
||||
| Wan2.2 T2V | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` |
|
||||
| LTX-2 | `Lightricks/LTX-2` |
|
||||
| Helios (Base / Mid / Distilled) | `BestWishYsh/Helios-Base`, `Helios-Mid`, `Helios-Distilled` |
|
||||
|
||||
## Wan2.2 T2V
|
||||
|
||||
@@ -323,3 +324,45 @@ curl -sS -X POST http://localhost:8098/v1/videos \
|
||||
-F "guidance_scale=3.0" \
|
||||
-F "seed=42"
|
||||
```
|
||||
|
||||
## Helios
|
||||
|
||||
Helios ships three variants (`Helios-Base`, `Helios-Mid`, `Helios-Distilled`) that
|
||||
share the same server launch. Variant-specific knobs (declared in
|
||||
`vllm_omni/model_extras/helios.py`) are sent per request through the generic
|
||||
`extra_params` JSON form field — no per-model server flags required.
|
||||
|
||||
### Start Server
|
||||
|
||||
```bash
|
||||
vllm serve BestWishYsh/Helios-Base --omni --port 8098
|
||||
# or: MODEL=BestWishYsh/Helios-Mid bash run_server_helios.sh
|
||||
```
|
||||
|
||||
### Send Requests (curl)
|
||||
|
||||
```bash
|
||||
# Helios-Base (Stage 1 only)
|
||||
bash run_curl_helios.sh
|
||||
|
||||
# Helios-Mid (Stage 2 pyramid + CFG-Zero*)
|
||||
PRESET=mid-stage2 MODEL=BestWishYsh/Helios-Mid bash run_curl_helios.sh
|
||||
|
||||
# Helios-Distilled (Stage 2 pyramid + DMD, few-step)
|
||||
PRESET=distilled MODEL=BestWishYsh/Helios-Distilled bash run_curl_helios.sh
|
||||
```
|
||||
|
||||
The `mid-stage2` and `distilled` presets attach an `extra_params` field, e.g. for Helios-Distilled:
|
||||
|
||||
```bash
|
||||
curl -sS -X POST http://localhost:8098/v1/videos \
|
||||
-H "Accept: application/json" \
|
||||
-F "prompt=A dynamic time-lapse of scenery rushing past the window of a speeding train." \
|
||||
-F "model=BestWishYsh/Helios-Distilled" \
|
||||
-F "size=640x384" \
|
||||
-F "num_frames=99" \
|
||||
-F "fps=16" \
|
||||
-F "guidance_scale=1.0" \
|
||||
-F "seed=42" \
|
||||
-F 'extra_params={"is_enable_stage2": true, "pyramid_num_inference_steps_list": [2, 2, 2], "is_amplify_first_chunk": true}'
|
||||
```
|
||||
@@ -0,0 +1,93 @@
|
||||
#!/bin/bash
|
||||
# Helios text-to-video curl example using the async video job API.
|
||||
#
|
||||
# Helios-specific knobs (declared in vllm_omni/model_extras/helios.py) are passed
|
||||
# through the generic `extra_params` JSON form field. Select a variant via PRESET:
|
||||
# PRESET=base -> Helios-Base, Stage 1 only (default)
|
||||
# PRESET=mid-stage2 -> Helios-Mid, Stage 2 pyramid + CFG-Zero*
|
||||
# PRESET=distilled -> Helios-Distilled, Stage 2 pyramid + DMD (few-step)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${BASE_URL:-http://localhost:8098}"
|
||||
MODEL="${MODEL:-BestWishYsh/Helios-Base}"
|
||||
PROMPT="${PROMPT:-A dynamic time-lapse of scenery rushing past the window of a speeding train.}"
|
||||
PRESET="${PRESET:-base}"
|
||||
OUTPUT_PATH="${OUTPUT_PATH:-helios_t2v_${PRESET}.mp4}"
|
||||
POLL_INTERVAL="${POLL_INTERVAL:-2}"
|
||||
|
||||
case "${PRESET}" in
|
||||
base)
|
||||
EXTRA_PARAMS=""
|
||||
GUIDANCE_SCALE="5.0"
|
||||
;;
|
||||
mid-stage2)
|
||||
EXTRA_PARAMS='{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [20, 20, 20], "use_cfg_zero_star": true, "use_zero_init": true, "zero_steps": 1}'
|
||||
GUIDANCE_SCALE="5.0"
|
||||
;;
|
||||
distilled)
|
||||
EXTRA_PARAMS='{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [2, 2, 2], "is_amplify_first_chunk": true}'
|
||||
GUIDANCE_SCALE="1.0"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown PRESET '${PRESET}' (expected base|mid-stage2|distilled)"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
create_args=(
|
||||
-sS -X POST "${BASE_URL}/v1/videos"
|
||||
-H "Accept: application/json"
|
||||
-F "prompt=${PROMPT}"
|
||||
-F "model=${MODEL}"
|
||||
-F "size=640x384"
|
||||
-F "num_frames=99"
|
||||
-F "fps=16"
|
||||
-F "num_inference_steps=50"
|
||||
-F "guidance_scale=${GUIDANCE_SCALE}"
|
||||
-F "seed=42"
|
||||
)
|
||||
if [ -n "${EXTRA_PARAMS}" ]; then
|
||||
create_args+=(-F "extra_params=${EXTRA_PARAMS}")
|
||||
fi
|
||||
|
||||
create_response=$(curl "${create_args[@]}")
|
||||
|
||||
video_id="$(echo "${create_response}" | jq -r '.id')"
|
||||
if [ -z "${video_id}" ] || [ "${video_id}" = "null" ]; then
|
||||
echo "Failed to create video job:"
|
||||
echo "${create_response}" | jq .
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Created video job ${video_id} (preset=${PRESET})"
|
||||
echo "${create_response}" | jq .
|
||||
|
||||
while true; do
|
||||
status_response="$(curl -sS "${BASE_URL}/v1/videos/${video_id}")"
|
||||
status="$(echo "${status_response}" | jq -r '.status')"
|
||||
|
||||
case "${status}" in
|
||||
queued|in_progress)
|
||||
echo "Video job ${video_id} status: ${status}"
|
||||
sleep "${POLL_INTERVAL}"
|
||||
;;
|
||||
completed)
|
||||
echo "${status_response}" | jq .
|
||||
break
|
||||
;;
|
||||
failed)
|
||||
echo "Video generation failed:"
|
||||
echo "${status_response}" | jq .
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
echo "Unexpected status response:"
|
||||
echo "${status_response}" | jq .
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
curl -sS -L "${BASE_URL}/v1/videos/${video_id}/content" -o "${OUTPUT_PATH}"
|
||||
echo "Saved video to ${OUTPUT_PATH}"
|
||||
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
# Helios online serving startup script.
|
||||
# All three variants (Helios-Base / Helios-Mid / Helios-Distilled) share the same
|
||||
# server launch; variant-specific knobs are sent per-request via `extra_params`
|
||||
# (see run_curl_helios.sh).
|
||||
|
||||
MODEL="${MODEL:-BestWishYsh/Helios-Base}"
|
||||
PORT="${PORT:-8098}"
|
||||
|
||||
echo "Starting Helios server..."
|
||||
echo "Model: $MODEL"
|
||||
echo "Port: $PORT"
|
||||
|
||||
vllm serve "$MODEL" --omni \
|
||||
--port "$PORT"
|
||||
+18
-25
@@ -4,27 +4,27 @@
|
||||
|
||||
- Vendor: Helios
|
||||
- Model: `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled`
|
||||
- Task: Text-to-video, image-to-video, and video-to-video generation
|
||||
- Mode: Offline inference with the bundled Helios example
|
||||
- Task: Text-to-video generation
|
||||
- Mode: Offline inference with the shared text_to_video example
|
||||
- Maintainer: Community
|
||||
|
||||
## When to use this recipe
|
||||
|
||||
Use this recipe when you want a known-good starting point for running Helios
|
||||
video generation with vLLM-Omni. The concrete command below focuses on
|
||||
`BestWishYsh/Helios-Base` text-to-video generation on one NVIDIA H20 GPU. The
|
||||
same example script also supports Helios-Mid, Helios-Distilled, image-to-video,
|
||||
and video-to-video; see the linked example docs for those model-specific flags.
|
||||
`BestWishYsh/Helios-Base` text-to-video generation on one NVIDIA H20 GPU via the
|
||||
shared `text_to_video.py` example. The same example also supports Helios-Mid and
|
||||
Helios-Distilled through the generic `--extra-body` flag (see below).
|
||||
Image-to-video and video-to-video require image/video conditioning inputs and
|
||||
are not covered by the text-to-video example.
|
||||
|
||||
## References
|
||||
|
||||
- Upstream repository: <https://github.com/PKU-YuanGroup/Helios>
|
||||
- Model weights:
|
||||
<https://huggingface.co/BestWishYsh/Helios-Base>
|
||||
- Upstream or canonical docs:
|
||||
[`docs/user_guide/examples/offline_inference/helios.md`](../../docs/user_guide/examples/offline_inference/helios.md)
|
||||
- Related example under `examples/`:
|
||||
[`examples/offline_inference/helios/README.md`](../../examples/offline_inference/helios/README.md)
|
||||
[`examples/offline_inference/text_to_video/text_to_video.md`](../../examples/offline_inference/text_to_video/text_to_video.md)
|
||||
|
||||
## Hardware Support
|
||||
|
||||
@@ -46,11 +46,10 @@ and video-to-video; see the linked example docs for those model-specific flags.
|
||||
Run the baseline Helios-Base text-to-video example from the repository root:
|
||||
|
||||
```bash
|
||||
cd examples/offline_inference/helios
|
||||
cd examples/offline_inference/text_to_video
|
||||
|
||||
python end2end.py \
|
||||
python text_to_video.py \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_t2v_base.mp4
|
||||
@@ -59,13 +58,12 @@ python end2end.py \
|
||||
To use cache-dit acceleration, run on a vLLM-Omni checkout that includes Helios cache-dit support and enable the cache backend:
|
||||
|
||||
```bash
|
||||
cd examples/offline_inference/helios
|
||||
cd examples/offline_inference/text_to_video
|
||||
|
||||
python end2end.py \
|
||||
python text_to_video.py \
|
||||
--cache-backend cache_dit \
|
||||
--enable-cache-dit-summary \
|
||||
--model BestWishYsh/Helios-Base \
|
||||
--sample-type t2v \
|
||||
--prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
|
||||
--guidance-scale 5.0 \
|
||||
--output helios_t2v_base.mp4
|
||||
@@ -85,22 +83,17 @@ Saved generated video to helios_t2v_base.mp4
|
||||
|
||||
- `--model BestWishYsh/Helios-Base` selects the base Helios checkpoint used by
|
||||
this recipe.
|
||||
- `--sample-type t2v` selects text-to-video generation. Use `i2v` with
|
||||
`--image-path` for image-to-video, or `v2v` with `--video-path` for
|
||||
video-to-video.
|
||||
- `--guidance-scale 5.0` matches the Helios-Base recommendation. Use
|
||||
`--guidance-scale 1.0` for Helios-Distilled.
|
||||
- `--cache-backend cache_dit` enables the cache-dit acceleration path.
|
||||
- `--enable-cache-dit-summary` prints cache-dit summary information after
|
||||
diffusion forward passes.
|
||||
- No separate stage config is required for this offline recipe; the bundled
|
||||
Helios example configures the pipeline through `end2end.py` arguments.
|
||||
- For Helios-Mid, add `--is-enable-stage2`,
|
||||
`--pyramid-num-inference-steps-list 20 20 20`, `--use-cfg-zero-star`,
|
||||
`--use-zero-init`, and `--zero-steps 1`.
|
||||
- For Helios-Distilled, add `--is-enable-stage2`,
|
||||
`--pyramid-num-inference-steps-list 2 2 2`, and
|
||||
`--is-amplify-first-chunk`.
|
||||
- No separate stage config is required for this offline recipe; the shared
|
||||
`text_to_video.py` example configures the pipeline through its arguments.
|
||||
- Helios-specific knobs (declared in `vllm_omni/model_extras/helios.py`) are
|
||||
passed via the generic `--extra-body` JSON flag:
|
||||
- Helios-Mid: `--extra-body '{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [20, 20, 20], "use_cfg_zero_star": true, "use_zero_init": true, "zero_steps": 1}'`
|
||||
- Helios-Distilled: `--extra-body '{"is_enable_stage2": true, "pyramid_num_inference_steps_list": [2, 2, 2], "is_amplify_first_chunk": true}'`
|
||||
|
||||
#### Known limitations
|
||||
|
||||
|
||||
@@ -57,6 +57,36 @@ def test_sensenova_extra_registry_declares_request_and_response_params() -> None
|
||||
assert should_init_extra_args_for_non_diffusion_stages("SenseNovaU1Pipeline") is False
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_helios_extra_registry_declares_request_and_response_params() -> None:
|
||||
expected_body = frozenset(
|
||||
{
|
||||
"is_enable_stage2",
|
||||
"pyramid_num_stages",
|
||||
"pyramid_num_inference_steps_list",
|
||||
"is_amplify_first_chunk",
|
||||
"is_skip_first_chunk",
|
||||
"use_cfg_zero_star",
|
||||
"use_zero_init",
|
||||
"zero_steps",
|
||||
"image",
|
||||
"video",
|
||||
"add_noise_to_image_latents",
|
||||
"image_noise_sigma_min",
|
||||
"image_noise_sigma_max",
|
||||
"add_noise_to_video_latents",
|
||||
"video_noise_sigma_min",
|
||||
"video_noise_sigma_max",
|
||||
}
|
||||
)
|
||||
# Both the base and pyramid class names resolve to the same declaration.
|
||||
for cls in ("HeliosPipeline", "HeliosPyramidPipeline"):
|
||||
assert get_extra_body_params(cls) == expected_body
|
||||
assert get_extra_output_params(cls) == frozenset()
|
||||
assert should_init_extra_args_for_non_diffusion_stages(cls) is False
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_unknown_pipeline_has_empty_extra_registry() -> None:
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Helios is a text/image/video -> video model. Its model-specific knobs are read
|
||||
# from sampling_params.extra_args in pipeline_helios.py; declaring them here
|
||||
# routes request `extra_body` fields into OmniDiffusionSamplingParams.extra_args
|
||||
# so the model can be driven through the standard task example.
|
||||
HELIOS_EXTRA_BODY_PARAMS = frozenset(
|
||||
{
|
||||
# pyramid / staged sampling
|
||||
"is_enable_stage2",
|
||||
"pyramid_num_stages",
|
||||
"pyramid_num_inference_steps_list",
|
||||
"is_amplify_first_chunk",
|
||||
"is_skip_first_chunk",
|
||||
# CFG-zero* controls
|
||||
"use_cfg_zero_star",
|
||||
"use_zero_init",
|
||||
"zero_steps",
|
||||
# conditioning inputs
|
||||
"image",
|
||||
"video",
|
||||
"add_noise_to_image_latents",
|
||||
"image_noise_sigma_min",
|
||||
"image_noise_sigma_max",
|
||||
"add_noise_to_video_latents",
|
||||
"video_noise_sigma_min",
|
||||
"video_noise_sigma_max",
|
||||
}
|
||||
)
|
||||
HELIOS_EXTRA_OUTPUT_PARAMS = frozenset()
|
||||
@@ -19,6 +19,10 @@ from vllm_omni.model_extras.bagel import (
|
||||
from vllm_omni.model_extras.bagel import (
|
||||
build_text_to_image_prompt as build_bagel_text_to_image_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras.helios import (
|
||||
HELIOS_EXTRA_BODY_PARAMS,
|
||||
HELIOS_EXTRA_OUTPUT_PARAMS,
|
||||
)
|
||||
from vllm_omni.model_extras.sensenova_u1 import (
|
||||
SENSENOVA_U1_EXTRA_BODY_PARAMS,
|
||||
SENSENOVA_U1_EXTRA_OUTPUT_PARAMS,
|
||||
@@ -76,6 +80,14 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
|
||||
"extra_body_params": SENSENOVA_U1_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": SENSENOVA_U1_EXTRA_OUTPUT_PARAMS,
|
||||
},
|
||||
"HeliosPipeline": {
|
||||
"extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,
|
||||
},
|
||||
"HeliosPyramidPipeline": {
|
||||
"extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user