[Frontend] Add MiniMax-H3 references and Ref2VA workflow (#7483)
Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
This commit is contained in:
Linze Shi authored and GitHub committed
1 parent
5f7f62e13c
commit
e78a5d01f7
9 files changed
+1235
-66
No files matched your search
@@ -139,6 +139,80 @@ You can configure per-stage sampling parameters for multi-stage models.
|
||||
>
|
||||
> Do not use `frame` and `references` together. Task routing is automatic from which inputs you connect.
|
||||
|
||||
For MiniMax-H3 Ref2VA, **Video References** accepts up to 9 images (`image_1`–`image_9`),
|
||||
3 videos (`video_1`–`video_3`), and 3 audio clips (`audio_1`–`audio_3`), with at most
|
||||
12 connected inputs in total. Any mixture containing at least one image or video
|
||||
is supported; empty and audio-only references are rejected. For example, you can
|
||||
combine 6 images, 3 videos, and 3 audio clips in one request.
|
||||
|
||||
Within each media type, references follow slot-number order, skipping unconnected
|
||||
slots. For example, connecting `image_2` and `image_9` sends `image_2` as the first
|
||||
image and `image_9` as the second. Each image slot uses the first image in its batch.
|
||||
Existing connections to `image_1`, `image_2`, `audio_1`, `audio_2`, `video_1`, and
|
||||
`video_2` remain valid in saved workflows.
|
||||
|
||||
#### MiniMax-H3 Reference to Video
|
||||
|
||||
Load [MiniMax-H3 Reference to Video](example_workflows/vLLM-Omni%20MiniMax-H3%20Reference%20to%20Video.json)
|
||||
from **Templates → ComfyUI-vLLM-Omni**, or drag the JSON onto the canvas.
|
||||
It adapts the [official ComfyUI R2V workflow](https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_r2v.json)
|
||||
to remote vLLM-Omni execution. Model loading, sampling, and VAE decoding run on
|
||||
the server; ComfyUI loads references and saves the returned video.
|
||||
|
||||
Use an extension version containing the audio-preserving output fix from
|
||||
[#7456](https://github.com/vllm-project/vllm-omni/pull/7456). Older versions can
|
||||
drop the server's audio while decoding the response.
|
||||
|
||||
1. Start a Ref2VA-capable service using the [MiniMax-H3 recipe](../../recipes/MiniMaxAI/MiniMax-H3.md).
|
||||
Set **Generate Video** to its `/v1` URL and served model name.
|
||||
2. Select your own image in **Load Image**. The template's filenames are placeholders
|
||||
relative to ComfyUI's input directory; no sample assets or model weights are bundled.
|
||||
3. Connect **Load Video** and **Load Audio** when needed. Duplicate the loaders to
|
||||
fill more reference slots, or disconnect the image when using video-only input.
|
||||
Keep the references connected to **Generate Video** and leave `frame` disconnected.
|
||||
4. Match prompt tags such as `<Picture 1>`, `<Video 1>`, and `<Audio 1>` to the
|
||||
connected references, counting each media type separately and skipping empty slots.
|
||||
5. Run the workflow. **Save Video** writes an MP4 under `output/video/` and preserves
|
||||
the generated audio when the audio-output prerequisite is installed.
|
||||
|
||||
The default is **1344×768, 24 FPS, 124 frames** (about 5.17 seconds), 50 sampling
|
||||
points, video flow shift 12, audio flow shift 3, and seed 42. The duration widget
|
||||
is set to 5.167 seconds, which converts to 124 frames at 24 FPS. Other H3 canvas presets
|
||||
are 1024×768, 768×768, 768×1024, and 768×1344. Keep frame counts at `17k+5` within
|
||||
the 4–15 second output range; examples are 107, 124, 209, and 345 frames.
|
||||
Reference video/audio clips must each be 2–15 seconds, with at most 15 seconds of
|
||||
reference video and at most 15 seconds of standalone reference audio per request.
|
||||
|
||||
For **Ref2VA Turbo**, use the same template. Configure a Ref2VA-only server with
|
||||
`--lora-backend peft --lora-path "$TURBO_LORA" --task-type ref2va`, following the
|
||||
recipe's hardware setup. For `minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors`,
|
||||
set **LoRA** `local_path` to that file's path on the server and connect it to
|
||||
**Generate Video**. Use `num_inference_steps=5`, `flow_shift=12`,
|
||||
`audio_flow_shift=3`, and LoRA `scale=1`. The API counts sigma points, so five
|
||||
points produce four denoiser evaluations. Use the Diffusers artifact, not its
|
||||
`_comfyui_` export. To return to base mode, disconnect LoRA and restore 50 steps.
|
||||
The 8-step v1.0 Ref2VA adapter instead requires nine sampling points and video
|
||||
flow shift 6. FastH3 is a separate startup-fused T2VA deployment.
|
||||
|
||||
Validate the template's wiring and defaults locally with:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/e2e/features/comfyui/test_h3_reference_workflow.py -q
|
||||
```
|
||||
|
||||
For a real-model check, run the workflow against H3 and inspect the saved file:
|
||||
|
||||
```bash
|
||||
ffprobe -v error -show_entries stream=codec_type,width,height,r_frame_rate,sample_rate,channels \
|
||||
-show_entries format=duration -of json "$OUTPUT_MP4"
|
||||
```
|
||||
|
||||
Check for 24 FPS video, stereo audio, the requested canvas, and aligned duration;
|
||||
also play the result to assess reference conditioning and audio/video synchronization.
|
||||
Record the server commit, model/adapter, input assets, prompt, settings, and output
|
||||
alongside the result. Local schema or mocked-server checks do not establish H3
|
||||
generation quality or replace this real-model validation.
|
||||
|
||||
#### H3 video upscale (WF-07)
|
||||
|
||||
The **vLLM-Omni MiniMax H3 Video Upscale** template generates video remotely, upscales it with SeedVR2, and saves the original and upscaled videos with the generated audio and FPS. See [workflow setup](docs/wf07-h3-upscale.md).
|
||||
|
||||
@@ -10,6 +10,9 @@ from .utils.api_client import VLLMOmniClient
|
||||
from .utils.logger import get_logger
|
||||
from .utils.models import lookup_model_spec
|
||||
from .utils.types import (
|
||||
MAX_REFERENCE_AUDIOS,
|
||||
MAX_REFERENCE_IMAGES,
|
||||
MAX_REFERENCE_VIDEOS,
|
||||
AudioFormat,
|
||||
AutoregressionSamplingParams,
|
||||
DiffusionSamplingParams,
|
||||
@@ -918,6 +921,10 @@ class VLLMOmniVideoReferences:
|
||||
"audio_2": ("AUDIO",),
|
||||
"video_1": ("VIDEO",),
|
||||
"video_2": ("VIDEO",),
|
||||
# Append ports to preserve connections in saved workflows.
|
||||
**{f"image_{i}": ("IMAGE",) for i in range(3, MAX_REFERENCE_IMAGES + 1)},
|
||||
**{f"audio_{i}": ("AUDIO",) for i in range(3, MAX_REFERENCE_AUDIOS + 1)},
|
||||
**{f"video_{i}": ("VIDEO",) for i in range(3, MAX_REFERENCE_VIDEOS + 1)},
|
||||
},
|
||||
}
|
||||
|
||||
@@ -934,21 +941,26 @@ class VLLMOmniVideoReferences:
|
||||
audio_2: AudioInput | None = None,
|
||||
video_1: VideoInput | None = None,
|
||||
video_2: VideoInput | None = None,
|
||||
image_3: torch.Tensor | None = None,
|
||||
image_4: torch.Tensor | None = None,
|
||||
image_5: torch.Tensor | None = None,
|
||||
image_6: torch.Tensor | None = None,
|
||||
image_7: torch.Tensor | None = None,
|
||||
image_8: torch.Tensor | None = None,
|
||||
image_9: torch.Tensor | None = None,
|
||||
audio_3: AudioInput | None = None,
|
||||
video_3: VideoInput | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
refs = VideoReferences()
|
||||
if image_1 is not None:
|
||||
refs["image_1"] = image_1
|
||||
if image_2 is not None:
|
||||
refs["image_2"] = image_2
|
||||
if audio_1 is not None:
|
||||
refs["audio_1"] = audio_1
|
||||
if audio_2 is not None:
|
||||
refs["audio_2"] = audio_2
|
||||
if video_1 is not None:
|
||||
refs["video_1"] = video_1
|
||||
if video_2 is not None:
|
||||
refs["video_2"] = video_2
|
||||
for kind, values in (
|
||||
("image", (image_1, image_2, image_3, image_4, image_5, image_6, image_7, image_8, image_9)),
|
||||
("video", (video_1, video_2, video_3)),
|
||||
("audio", (audio_1, audio_2, audio_3)),
|
||||
):
|
||||
for index, value in enumerate(values, start=1):
|
||||
if value is not None:
|
||||
refs[f"{kind}_{index}"] = value
|
||||
return (refs,)
|
||||
@@ -20,6 +20,7 @@ from comfy_api.input import AudioInput, VideoInput
|
||||
|
||||
from .format import (
|
||||
audio_to_base64,
|
||||
audio_to_bytes,
|
||||
base64_to_audio,
|
||||
base64_to_image_tensor,
|
||||
bytes_to_audio,
|
||||
@@ -31,7 +32,13 @@ from .format import (
|
||||
)
|
||||
from .logger import get_logger, pretty_printer
|
||||
from .models import lookup_model_spec
|
||||
from .types import AudioFormat
|
||||
from .types import (
|
||||
MAX_REFERENCE_AUDIOS,
|
||||
MAX_REFERENCE_IMAGES,
|
||||
MAX_REFERENCE_VIDEOS,
|
||||
MAX_TOTAL_REFERENCES,
|
||||
AudioFormat,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -304,38 +311,44 @@ class VLLMOmniClient:
|
||||
|
||||
# === multimodal inputs (first-last-frames, references, etc.) ===
|
||||
input_reference_image: torch.Tensor | None = None
|
||||
audio_reference: AudioInput | None = None
|
||||
reference_videos: list[VideoInput] = []
|
||||
video_task: str | None = None
|
||||
|
||||
if frame is not None:
|
||||
input_reference_image = frame
|
||||
video_task = "fl2va"
|
||||
elif references is not None:
|
||||
images = [references[k] for k in ("image_1", "image_2") if k in references and references[k] is not None]
|
||||
audios = [references[k] for k in ("audio_1", "audio_2") if k in references and references[k] is not None]
|
||||
videos = [references[k] for k in ("video_1", "video_2") if k in references and references[k] is not None]
|
||||
|
||||
if not images and not audios and not videos:
|
||||
raise ValueError("references is empty; connect at least one image/audio/video.")
|
||||
|
||||
if videos:
|
||||
if images or audios:
|
||||
raise ValueError(
|
||||
"Video references cannot be combined with image or audio references. "
|
||||
"Connect only video_1/video_2 for multi-video Ref2VA."
|
||||
)
|
||||
reference_videos = videos
|
||||
video_task = "ref2va"
|
||||
elif len(images) == 1 and len(audios) == 1:
|
||||
input_reference_image = images[0]
|
||||
audio_reference = audios[0]
|
||||
video_task = "ref2va"
|
||||
else:
|
||||
reference_formats = (
|
||||
("image", MAX_REFERENCE_IMAGES, "png", "image/png", image_tensor_to_png_bytes),
|
||||
("video", MAX_REFERENCE_VIDEOS, "mp4", "video/mp4", video_to_bytes),
|
||||
("audio", MAX_REFERENCE_AUDIOS, "mp3", "audio/mpeg", audio_to_bytes),
|
||||
)
|
||||
supported_inputs = {f"{kind}_{i}" for kind, limit, *_ in reference_formats for i in range(1, limit + 1)}
|
||||
connected = {name: value for name, value in references.items() if value is not None}
|
||||
unsupported = connected.keys() - supported_inputs
|
||||
if unsupported:
|
||||
raise ValueError(f"Unsupported reference input(s): {', '.join(sorted(unsupported))}.")
|
||||
if not any(name.startswith(("image_", "video_")) for name in connected):
|
||||
raise ValueError(
|
||||
"Invalid references combination. Supported modes: "
|
||||
"(1) one or more videos only, or (2) exactly one image and one audio."
|
||||
"references requires at least one image or video; audio-only inputs are not supported."
|
||||
)
|
||||
if len(connected) > MAX_TOTAL_REFERENCES:
|
||||
raise ValueError(
|
||||
f"references supports at most {MAX_TOTAL_REFERENCES} inputs in total "
|
||||
f"(up to {MAX_REFERENCE_IMAGES} images, {MAX_REFERENCE_VIDEOS} videos, "
|
||||
f"and {MAX_REFERENCE_AUDIOS} audios)."
|
||||
)
|
||||
for kind, limit, extension, content_type, encode in reference_formats:
|
||||
for index in range(1, limit + 1):
|
||||
name = f"{kind}_{index}"
|
||||
if name in connected:
|
||||
filename = f"{name}.{extension}"
|
||||
form.add_field(
|
||||
"input_references",
|
||||
encode(connected[name], filename),
|
||||
filename=filename,
|
||||
content_type=content_type,
|
||||
)
|
||||
video_task = "ref2va"
|
||||
else:
|
||||
video_task = "t2va"
|
||||
|
||||
@@ -348,21 +361,6 @@ class VLLMOmniClient:
|
||||
content_type="image/png",
|
||||
)
|
||||
|
||||
if audio_reference is not None:
|
||||
form.add_field(
|
||||
"audio_reference",
|
||||
json.dumps({"audio_url": audio_to_base64(audio_reference)}, ensure_ascii=False),
|
||||
)
|
||||
|
||||
for idx, video in enumerate(reference_videos, start=1):
|
||||
video_filename = f"reference_{idx}.mp4"
|
||||
form.add_field(
|
||||
"input_references",
|
||||
video_to_bytes(video, video_filename),
|
||||
filename=video_filename,
|
||||
content_type="video/mp4",
|
||||
)
|
||||
|
||||
# === model specific params. Either use a specialized builder, or add flattened fields as-is ===
|
||||
if model_params is not None:
|
||||
model_params = dict(model_params)
|
||||
|
||||
@@ -30,6 +30,12 @@ class MiniMaxH3ModelSpecificParams(dict):
|
||||
pass
|
||||
|
||||
|
||||
MAX_REFERENCE_IMAGES = 9
|
||||
MAX_REFERENCE_VIDEOS = 3
|
||||
MAX_REFERENCE_AUDIOS = 3
|
||||
MAX_TOTAL_REFERENCES = 12
|
||||
|
||||
|
||||
class VideoReferences(dict):
|
||||
pass
|
||||
|
||||
|
||||
+786
@@ -0,0 +1,786 @@
|
||||
{
|
||||
"id": "e7a4e606-6510-4c4e-b127-9a8b6bf3ea71",
|
||||
"revision": 0,
|
||||
"last_node_id": 11,
|
||||
"last_link_id": 5,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "LoadImage",
|
||||
"pos": [
|
||||
40,
|
||||
100
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
300
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "image",
|
||||
"name": "image",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "image"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "choose file to upload",
|
||||
"name": "upload",
|
||||
"type": "IMAGEUPLOAD",
|
||||
"widget": {
|
||||
"name": "upload"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "IMAGE",
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"localized_name": "MASK",
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"h3_reference.png",
|
||||
"image"
|
||||
],
|
||||
"title": "Reference image \u2014 select a local file"
|
||||
},
|
||||
{
|
||||
"type": "LoadVideo",
|
||||
"size": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "VIDEO",
|
||||
"type": "VIDEO",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"h3_reference.mp4"
|
||||
],
|
||||
"id": 2,
|
||||
"pos": [
|
||||
40,
|
||||
470
|
||||
],
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"title": "Optional reference video"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "LoadAudio",
|
||||
"pos": [
|
||||
40,
|
||||
1100
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "audio",
|
||||
"name": "audio",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "audio"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "audioUI",
|
||||
"name": "audioUI",
|
||||
"type": "AUDIO_UI",
|
||||
"widget": {
|
||||
"name": "audioUI"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "choose file to upload",
|
||||
"name": "upload",
|
||||
"type": "AUDIOUPLOAD",
|
||||
"widget": {
|
||||
"name": "upload"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "AUDIO",
|
||||
"name": "AUDIO",
|
||||
"type": "AUDIO",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadAudio"
|
||||
},
|
||||
"widgets_values": [
|
||||
"h3_reference.wav",
|
||||
null,
|
||||
""
|
||||
],
|
||||
"title": "Optional reference audio"
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "VLLMOmniVideoReferences",
|
||||
"pos": [
|
||||
420,
|
||||
100
|
||||
],
|
||||
"size": [
|
||||
300,
|
||||
365
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "image_1",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "image_2",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "audio_1",
|
||||
"type": "AUDIO",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "audio_2",
|
||||
"type": "AUDIO",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "video_1",
|
||||
"type": "VIDEO",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "video_2",
|
||||
"type": "VIDEO",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_3",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_4",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_5",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_6",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_7",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_8",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "image_9",
|
||||
"type": "IMAGE",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "audio_3",
|
||||
"type": "AUDIO",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "video_3",
|
||||
"type": "VIDEO",
|
||||
"shape": 7,
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "references",
|
||||
"name": "references",
|
||||
"type": "VIDEO_REFERENCES",
|
||||
"links": [
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniVideoReferences"
|
||||
},
|
||||
"widgets_values": [],
|
||||
"title": "References \u2014 up to 12 total"
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VLLMOmniDiffusionSampling",
|
||||
"pos": [
|
||||
420,
|
||||
530
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
275
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "n",
|
||||
"name": "n",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "n"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "num_inference_steps",
|
||||
"name": "num_inference_steps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "num_inference_steps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "guidance_scale",
|
||||
"name": "guidance_scale",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "guidance_scale"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "true_cfg_scale",
|
||||
"name": "true_cfg_scale",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "true_cfg_scale"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "vae_use_slicing",
|
||||
"name": "vae_use_slicing",
|
||||
"type": "BOOLEAN",
|
||||
"widget": {
|
||||
"name": "vae_use_slicing"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "vae_use_tiling",
|
||||
"name": "vae_use_tiling",
|
||||
"type": "BOOLEAN",
|
||||
"widget": {
|
||||
"name": "vae_use_tiling"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "seed",
|
||||
"name": "seed",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "seed"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "diffusion sampling params",
|
||||
"name": "diffusion sampling params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"links": [
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniDiffusionSampling"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
50,
|
||||
1.0,
|
||||
1.0,
|
||||
false,
|
||||
false,
|
||||
42,
|
||||
"fixed"
|
||||
],
|
||||
"title": "Sampling \u2014 base 50 / Turbo 5"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [
|
||||
790,
|
||||
750
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
105
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "audio_flow_shift",
|
||||
"name": "audio_flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_flow_shift"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "flow_shift",
|
||||
"name": "flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "flow_shift"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "MiniMax-H3 video params",
|
||||
"name": "MiniMax-H3 video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": [
|
||||
4
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniMiniMaxH3Params"
|
||||
},
|
||||
"widgets_values": [
|
||||
3.0,
|
||||
12.0
|
||||
],
|
||||
"title": "H3 video/audio flow shifts"
|
||||
},
|
||||
{
|
||||
"type": "VLLMOmniRemoteLoRA",
|
||||
"size": [
|
||||
310,
|
||||
180
|
||||
],
|
||||
"flags": {},
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "lora",
|
||||
"type": "REMOTE_LORA",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniRemoteLoRA"
|
||||
},
|
||||
"widgets_values": [
|
||||
"",
|
||||
"h3-ref2va-turbo",
|
||||
1.0,
|
||||
0
|
||||
],
|
||||
"id": 7,
|
||||
"pos": [
|
||||
420,
|
||||
940
|
||||
],
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"title": "Optional Turbo LoRA \u2014 connect to enable"
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [
|
||||
800,
|
||||
100
|
||||
],
|
||||
"size": [
|
||||
470,
|
||||
560
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "frame",
|
||||
"name": "frame",
|
||||
"shape": 7,
|
||||
"type": "IMAGE",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "references",
|
||||
"name": "references",
|
||||
"shape": 7,
|
||||
"type": "VIDEO_REFERENCES",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"localized_name": "sampling_params",
|
||||
"name": "sampling_params",
|
||||
"shape": 7,
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"link": 3
|
||||
},
|
||||
{
|
||||
"localized_name": "lora",
|
||||
"name": "lora",
|
||||
"shape": 7,
|
||||
"type": "REMOTE_LORA",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model_params",
|
||||
"name": "model_params",
|
||||
"shape": 7,
|
||||
"type": "VIDEO_PARAMS",
|
||||
"link": 4
|
||||
},
|
||||
{
|
||||
"localized_name": "url",
|
||||
"name": "url",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "url"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model",
|
||||
"name": "model",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "model"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "prompt",
|
||||
"name": "prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "negative_prompt",
|
||||
"name": "negative_prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "negative_prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "width",
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "height",
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "fps",
|
||||
"name": "fps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "fps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "duration",
|
||||
"name": "duration",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "duration"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
5
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniGenerateVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"http://localhost:8000/v1",
|
||||
"MiniMaxAI/MiniMax-H3",
|
||||
"Use the subject and visual style from <Picture 1>. Create a cinematic shot with a gentle camera move and natural ambient sound.",
|
||||
"",
|
||||
1344,
|
||||
768,
|
||||
24,
|
||||
5.167
|
||||
],
|
||||
"title": "MiniMax-H3 Reference to Video"
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "SaveVideo",
|
||||
"pos": [
|
||||
1360,
|
||||
100
|
||||
],
|
||||
"size": [
|
||||
410,
|
||||
420
|
||||
],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"link": 5
|
||||
},
|
||||
{
|
||||
"localized_name": "filename_prefix",
|
||||
"name": "filename_prefix",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "filename_prefix"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "format",
|
||||
"name": "format",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "format"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "codec",
|
||||
"name": "codec",
|
||||
"type": "COMFY_DYNAMICCOMBO_V3",
|
||||
"widget": {
|
||||
"name": "codec"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"video/MiniMax-H3-Ref2VA",
|
||||
"mp4",
|
||||
"h264"
|
||||
],
|
||||
"title": "Save generated video with audio"
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
1360,
|
||||
590
|
||||
],
|
||||
"size": [
|
||||
440,
|
||||
595
|
||||
],
|
||||
"flags": {},
|
||||
"order": 9,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "Reference inputs and output settings",
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"## Reference to Video\nSelect an image, set the server URL and served model name, then run.\n\nConnect the optional video/audio loaders to Video References, or duplicate loaders for more inputs: up to 9 images, 3 videos, 3 audio clips, 12 total. At least one image or video is required.\n\nSlots are numbered separately per media type. Empty slots are skipped. Update prompt tags such as <Picture 1>, <Video 1>, <Audio 1> to match the connected references. Videos keep their own soundtrack in the reference upload.\n\n**Output:** 24 FPS; 124 frames (~5.17 s). Valid frame counts are 107, 124, 141, 158, 175, 192, 209, 226, 243, 260, 277, 294, 311, 328, 345.\n\n**Width x height:** 1344x768 (16:9), 1024x768 (4:3), 768x768 (1:1), 768x1024 (3:4), 768x1344 (9:16).\n\nEach reference video/audio clip must be 2\u201315 s. Total reference-video duration <=15 s; total standalone-audio duration <=15 s.\n\nRequires the expanded Video References node and the audio-preserving video output fix. See the extension README for setup and validation status."
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
790,
|
||||
940
|
||||
],
|
||||
"size": [
|
||||
480,
|
||||
455
|
||||
],
|
||||
"flags": {},
|
||||
"order": 10,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "Optional Turbo \u2014 same workflow",
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"## Turbo Ref2VA\nBase mode: leave LoRA disconnected; use 50 steps, video flow shift 12, audio flow shift 3.\n\nFor the 4-step v0.1 Ref2VA Turbo adapter:\n1. Start a Ref2VA-only server with the Diffusers adapter loaded (see README).\n2. Set LoRA local_path to its path on the server, then connect LoRA to Generate Video.\n3. Set Sampling num_inference_steps to **5**; keep shifts at **12 / 3**, scale **1**.\n\nvLLM-Omni counts sigma points: 5 points give 4 denoiser evaluations. Use minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors, not the _comfyui_ export.\n\nDisconnect LoRA and restore 50 steps to return to base mode. FastH3 is a separate T2VA deployment, not this option.\n\nAdapted from the official ComfyUI MiniMax-H3 R2V workflow: https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_r2v.json"
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
2,
|
||||
4,
|
||||
0,
|
||||
8,
|
||||
1,
|
||||
"VIDEO_REFERENCES"
|
||||
],
|
||||
[
|
||||
3,
|
||||
5,
|
||||
0,
|
||||
8,
|
||||
2,
|
||||
"SAMPLING_PARAMS"
|
||||
],
|
||||
[
|
||||
4,
|
||||
6,
|
||||
0,
|
||||
8,
|
||||
4,
|
||||
"VIDEO_PARAMS"
|
||||
],
|
||||
[
|
||||
5,
|
||||
8,
|
||||
0,
|
||||
9,
|
||||
0,
|
||||
"VIDEO"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"workflowRendererVersion": "LG",
|
||||
"ds": {
|
||||
"scale": 0.85,
|
||||
"offset": [
|
||||
100,
|
||||
100
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
Reference in new issue
Block a user