Harden the shared video-audio example
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
65b3d41b3a
commit
6bdbaeb038
9 files changed
+533
-110
No files matched your search
@@ -3,7 +3,25 @@
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/x_to_video_audio>.
|
||||
|
||||
|
||||
The `DreamID-Omni` pipeline generates short videos from text, image and video.
|
||||
The shared runner generates video with optional audio from text, image, and audio
|
||||
inputs. It can be extended through `vllm_omni.model_extras` without adding
|
||||
another model-specific runner.
|
||||
|
||||
## Shared request and output contract
|
||||
|
||||
The example builds one canonical request envelope: `prompt` is text,
|
||||
`modalities` is `["video"]`, and optional `multi_modal_data` contains RGB PIL
|
||||
images plus audio `(waveform, sample_rate)` tuples. Audio files are loaded in
|
||||
full; the runner does not silently select or crop a time range. Prepare a
|
||||
trimmed input file explicitly when only part of a recording should condition
|
||||
the model.
|
||||
|
||||
Model-specific request translation (for example, DreamID-Omni's two negative
|
||||
prompts and its 16 kHz input requirement) is declared in `model_extras`. Model
|
||||
outputs must provide `fps` and, when audio is present, `audio_sample_rate`.
|
||||
`--fps` and `--audio-sample-rate` are available as explicit metadata overrides.
|
||||
The shared encoder normalizes supported video tensor layouts, so this runner
|
||||
does not branch on a model name or assume a model-specific frame layout.
|
||||
|
||||
## Local CLI Usage
|
||||
### Download the Model locally
|
||||
@@ -91,7 +109,8 @@ python x_to_video_audio.py \
|
||||
Key arguments:
|
||||
- `--prompt`: text description (string).
|
||||
- `--model`: path to the model local directory.
|
||||
- `--height/--width`: output resolution (defaults 704 * 1024).
|
||||
- `--model-class-name`: optional diffusion pipeline class override.
|
||||
- `--height/--width`: output resolution (defaults 704 * 1280).
|
||||
- `--image-path`: path to the input image list.
|
||||
- `--audio-path`: path to the input audio list, indicate the timbre of the output video.
|
||||
- `--cfg-parallel-size`: number of parallel cfg parallel (defaults 1).
|
||||
@@ -101,6 +120,9 @@ Key arguments:
|
||||
- `--num-inference-steps`: number of denoising steps (defaults 45).
|
||||
- `--video-negative-prompt`: negative prompt for video generation.
|
||||
- `--audio-negative-prompt`: negative prompt for audio generation.
|
||||
- `--extra-body`: JSON object containing parameters declared by the selected model.
|
||||
- `--fps`: explicit output FPS override when a model does not return that metadata.
|
||||
- `--audio-sample-rate`: explicit output audio sample-rate override.
|
||||
- `--enable-cpu-offload`: enable CPU offload (defaults False).
|
||||
|
||||
## Example materials
|
||||
|
||||
Reference in new issue
Block a user