[Frontend][Model] Add Cosmos3 control uploads to video API (#7027)

Signed-off-by: Fred Huang <frhuang@nvidia.com>
This commit is contained in:
FredHuangNV authored and GitHub committed 2026-09-07 01:38:56 -07:00
1 parent 677265b64b
commit 85b0bad2d3
6 files changed
+434 -9

No files matched your search

+19
View File
@@ -68,6 +68,8 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
| Parameter | Type | Default | Description |
| ----------- | ------ | --------- | ------------- |
| `input_reference` | file | null | Uploaded reference image or video for image-to-video/video-to-video requests |
| `control_reference` | file | null | Optional uploaded image/video control, up to 512 MiB, for models that declare control-upload support |
| `control_type` | string | null | Model control name associated with `control_reference`; currently Cosmos3 supports `edge`, `blur`, `depth`, `seg`, and `wsm` |
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` or `video_reference` |
| `video_reference` | string | null | JSON-encoded reference video payload; do not combine with `input_reference` or `image_reference` |
| `audio_reference` | string | null | JSON-encoded audio reference for speech-to-video: `{"audio_url": "..."}` — supports HTTP(s) URLs or base64 data URLs |
@@ -177,6 +179,23 @@ corresponding addition. `negative_metadata_mode` accepts `same`, `inverse`, or
`none` and defaults to `same` for transfer. See the Cosmos3 recipe for complete
examples.
A client that cannot place the control on the server filesystem can upload one
control with `control_reference` and identify it with `control_type`. The API
streams the upload to request-scoped storage, supplies its path to the model,
and removes it after synchronous or asynchronous generation completes. Other
options for the selected control can remain in `extra_params`; do not also set
`control` or `control_path` there. Uploads larger than 512 MiB are rejected.
```bash
curl -s http://localhost:8091/v1/videos/sync \
-F "prompt=Preserve the scene while following the world-state control" \
-F "input_reference=@input.mp4;type=video/mp4" \
-F "control_reference=@wsm.mp4;type=video/mp4" \
-F "control_type=wsm" \
-F 'extra_params={"wsm":{"control_weight":1.0}}' \
-o output.mp4
```
HTTP redirects for `image_reference.image_url` follow vLLM's
`VLLM_MEDIA_URL_ALLOW_REDIRECTS` setting. Before starting the server, set it to
`1` (the default) to allow redirects or `0` to reject them. A redirect target