Harden the shared video-audio example

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-17 13:48:27 +08:00
1 parent 65b3d41b3a
commit 6bdbaeb038
9 files changed
+533 -110

No files matched your search

@@ -3,7 +3,25 @@
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/x_to_video_audio>.
The `DreamID-Omni` pipeline generates short videos from text, image and video.
The shared runner generates video with optional audio from text, image, and audio
inputs. It can be extended through `vllm_omni.model_extras` without adding
another model-specific runner.
## Shared request and output contract
The example builds one canonical request envelope: `prompt` is text,
`modalities` is `["video"]`, and optional `multi_modal_data` contains RGB PIL
images plus audio `(waveform, sample_rate)` tuples. Audio files are loaded in
full; the runner does not silently select or crop a time range. Prepare a
trimmed input file explicitly when only part of a recording should condition
the model.
Model-specific request translation (for example, DreamID-Omni's two negative
prompts and its 16 kHz input requirement) is declared in `model_extras`. Model
outputs must provide `fps` and, when audio is present, `audio_sample_rate`.
`--fps` and `--audio-sample-rate` are available as explicit metadata overrides.
The shared encoder normalizes supported video tensor layouts, so this runner
does not branch on a model name or assume a model-specific frame layout.
## Local CLI Usage
### Download the Model locally
@@ -91,7 +109,8 @@ python x_to_video_audio.py \
Key arguments:
- `--prompt`: text description (string).
- `--model`: path to the model local directory.
- `--height/--width`: output resolution (defaults 704 * 1024).
- `--model-class-name`: optional diffusion pipeline class override.
- `--height/--width`: output resolution (defaults 704 * 1280).
- `--image-path`: path to the input image list.
- `--audio-path`: path to the input audio list, indicate the timbre of the output video.
- `--cfg-parallel-size`: number of parallel cfg parallel (defaults 1).
@@ -101,6 +120,9 @@ Key arguments:
- `--num-inference-steps`: number of denoising steps (defaults 45).
- `--video-negative-prompt`: negative prompt for video generation.
- `--audio-negative-prompt`: negative prompt for audio generation.
- `--extra-body`: JSON object containing parameters declared by the selected model.
- `--fps`: explicit output FPS override when a model does not return that metadata.
- `--audio-sample-rate`: explicit output audio sample-rate override.
- `--enable-cpu-offload`: enable CPU offload (defaults False).
## Example materials