Cosmos3 video to video generation (#4266)
Signed-off-by: Maciej Bala <mbala@nvidia.com>
This commit is contained in:
1 parent
9c9d4ed4a8
commit
2f25195750
14 files changed
+1037
-94
No files matched your search
@@ -67,8 +67,9 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `input_reference` | file | null | Uploaded reference image for image-to-video requests |
|
||||
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` |
|
||||
| `input_reference` | file | null | Uploaded reference image or video for image-to-video/video-to-video requests |
|
||||
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` or `video_reference` |
|
||||
| `video_reference` | string | null | JSON-encoded reference video payload; do not combine with `input_reference` or `image_reference` |
|
||||
| `width` | integer | model default | Output video width |
|
||||
| `height` | integer | model default | Output video height |
|
||||
| `num_frames` | integer | model default | Number of generated frames |
|
||||
@@ -80,6 +81,8 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
|
||||
| `flow_shift` | number | null | Scheduler flow-shift value |
|
||||
| `true_cfg_scale` | number | null | True CFG scale when supported by the model |
|
||||
| `seed` | integer | null | Random seed for reproducibility |
|
||||
| `generate_sound` | boolean | false | Request model-generated audio for video models that support sound generation |
|
||||
| `sound_duration` | number | null | Duration in seconds for generated audio; defaults to generated video duration |
|
||||
| `negative_prompt` | string | null | Text describing what to avoid in the generated video |
|
||||
| `enable_frame_interpolation` | boolean | null | Enable post-generation frame interpolation |
|
||||
| `frame_interpolation_exp` | integer | null | Interpolation exponent; `1=2x`, `2=4x`, and so on |
|
||||
@@ -123,6 +126,41 @@ curl -s http://localhost:8091/v1/videos \
|
||||
-F "fps=16"
|
||||
```
|
||||
|
||||
### Video-to-Video
|
||||
|
||||
For models that support video conditioning, upload the reference video with
|
||||
`input_reference`:
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8091/v1/videos \
|
||||
-F "prompt=continue this motion with consistent subjects and lighting" \
|
||||
-F "input_reference=@input.mp4;type=video/mp4" \
|
||||
-F "width=1280" \
|
||||
-F "height=720" \
|
||||
-F "num_frames=80" \
|
||||
-F "fps=16"
|
||||
```
|
||||
|
||||
You can also pass a JSON-safe video URL or `data:video/...;base64,...` payload
|
||||
through `video_reference`. Do not send `video_reference` together with
|
||||
`input_reference` or `image_reference`.
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8091/v1/videos \
|
||||
-F "prompt=continue this motion with consistent subjects and lighting" \
|
||||
-F 'video_reference={"video_url":"https://example.com/input.mp4"}' \
|
||||
-F "width=1280" \
|
||||
-F "height=720" \
|
||||
-F "num_frames=80" \
|
||||
-F "fps=16"
|
||||
```
|
||||
|
||||
JSON references currently support `image_url`/`video_url`; `file_id` references
|
||||
are not implemented yet. Models may expose additional V2V controls through
|
||||
`extra_params`. For example, Cosmos3 supports
|
||||
`condition_frame_indexes_vision` and `condition_video_keep` to select which
|
||||
decoded reference frames are used as clean conditioning.
|
||||
|
||||
### Synchronous Generation
|
||||
|
||||
```bash
|
||||
@@ -146,7 +184,10 @@ export VLLM_OMNI_STORAGE_PATH=/var/tmp/vllm-omni-videos
|
||||
|
||||
## Model-Specific Examples
|
||||
|
||||
For complete text-to-video and image-to-video walkthroughs, see:
|
||||
For complete text-to-video, image-to-video, and model-specific video-to-video
|
||||
walkthroughs, see:
|
||||
|
||||
- [Text-to-Video](../user_guide/examples/online_serving/text_to_video.md)
|
||||
- [Image-to-Video](../user_guide/examples/online_serving/image_to_video.md)
|
||||
- [Cosmos3 recipes](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md)
|
||||
for model-specific video-to-video examples and conditioning controls
|
||||
Reference in new issue
Block a user