Cosmos3 video to video generation (#4266)

Signed-off-by: Maciej Bala <mbala@nvidia.com>
This commit is contained in:
MaciejBalaNV authored and GitHub committed 2026-06-09 03:04:05 +08:00
1 parent 9c9d4ed4a8
commit 2f25195750
14 files changed
+1037 -94

No files matched your search

+44 -3
View File
@@ -67,8 +67,9 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input_reference` | file | null | Uploaded reference image for image-to-video requests |
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` |
| `input_reference` | file | null | Uploaded reference image or video for image-to-video/video-to-video requests |
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` or `video_reference` |
| `video_reference` | string | null | JSON-encoded reference video payload; do not combine with `input_reference` or `image_reference` |
| `width` | integer | model default | Output video width |
| `height` | integer | model default | Output video height |
| `num_frames` | integer | model default | Number of generated frames |
@@ -80,6 +81,8 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
| `flow_shift` | number | null | Scheduler flow-shift value |
| `true_cfg_scale` | number | null | True CFG scale when supported by the model |
| `seed` | integer | null | Random seed for reproducibility |
| `generate_sound` | boolean | false | Request model-generated audio for video models that support sound generation |
| `sound_duration` | number | null | Duration in seconds for generated audio; defaults to generated video duration |
| `negative_prompt` | string | null | Text describing what to avoid in the generated video |
| `enable_frame_interpolation` | boolean | null | Enable post-generation frame interpolation |
| `frame_interpolation_exp` | integer | null | Interpolation exponent; `1=2x`, `2=4x`, and so on |
@@ -123,6 +126,41 @@ curl -s http://localhost:8091/v1/videos \
-F "fps=16"
```
### Video-to-Video
For models that support video conditioning, upload the reference video with
`input_reference`:
```bash
curl -s http://localhost:8091/v1/videos \
-F "prompt=continue this motion with consistent subjects and lighting" \
-F "input_reference=@input.mp4;type=video/mp4" \
-F "width=1280" \
-F "height=720" \
-F "num_frames=80" \
-F "fps=16"
```
You can also pass a JSON-safe video URL or `data:video/...;base64,...` payload
through `video_reference`. Do not send `video_reference` together with
`input_reference` or `image_reference`.
```bash
curl -s http://localhost:8091/v1/videos \
-F "prompt=continue this motion with consistent subjects and lighting" \
-F 'video_reference={"video_url":"https://example.com/input.mp4"}' \
-F "width=1280" \
-F "height=720" \
-F "num_frames=80" \
-F "fps=16"
```
JSON references currently support `image_url`/`video_url`; `file_id` references
are not implemented yet. Models may expose additional V2V controls through
`extra_params`. For example, Cosmos3 supports
`condition_frame_indexes_vision` and `condition_video_keep` to select which
decoded reference frames are used as clean conditioning.
### Synchronous Generation
```bash
@@ -146,7 +184,10 @@ export VLLM_OMNI_STORAGE_PATH=/var/tmp/vllm-omni-videos
## Model-Specific Examples
For complete text-to-video and image-to-video walkthroughs, see:
For complete text-to-video, image-to-video, and model-specific video-to-video
walkthroughs, see:
- [Text-to-Video](../user_guide/examples/online_serving/text_to_video.md)
- [Image-to-Video](../user_guide/examples/online_serving/image_to_video.md)
- [Cosmos3 recipes](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md)
for model-specific video-to-video examples and conditioning controls