[Frontend] Add MiniMax H3 ComfyUI text-to-video workflow (WF-01) (#7423)
Signed-off-by: Hu <hu.zhu@connect.polyu.hk>
This commit is contained in:
4 files changed
+477
No files matched your search
@@ -213,6 +213,15 @@ Record the server commit, model/adapter, input assets, prompt, settings, and out
|
||||
alongside the result. Local schema or mocked-server checks do not establish H3
|
||||
generation quality or replace this real-model validation.
|
||||
|
||||
### MiniMax H3 Text to Video
|
||||
|
||||
Import [the H3 text-to-video template](example_workflows/MiniMax_H3_Text_to_Video.json)
|
||||
for remote video generation with audio. It includes connected base presets and
|
||||
optional Turbo sampling, H3 parameters, and Remote LoRA nodes. See the
|
||||
[workflow guide](docs/minimax-h3-t2v.md) for server setup, artifact-specific Turbo
|
||||
settings, and saved-video/audio validation. Set Remote LoRA’s `local_path` to
|
||||
the downloaded Turbo artifact on your server before enabling Turbo.
|
||||
|
||||
#### H3 video upscale (WF-07)
|
||||
|
||||
The **vLLM-Omni MiniMax H3 Video Upscale** template generates video remotely, upscales it with SeedVR2, and saves the original and upscaled videos with the generated audio and FPS. See [workflow setup](docs/wf07-h3-upscale.md).
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# MiniMax H3 Text to Video (WF-01)
|
||||
|
||||
Import [MiniMax_H3_Text_to_Video.json](../example_workflows/MiniMax_H3_Text_to_Video.json)
|
||||
into ComfyUI, or select it under **Templates → ComfyUI-vLLM-Omni**.
|
||||
The template connects existing nodes to a remote H3 service and saves the
|
||||
returned video with audio. ComfyUI does not load H3 weights.
|
||||
|
||||
## Setup
|
||||
|
||||
Install the [ComfyUI extension](../README.md#installation) and follow the
|
||||
[H3 deployment recipe](../../../recipes/MiniMaxAI/MiniMax-H3.md) for your hardware.
|
||||
For a two-GPU FL2VA server with sufficient memory:
|
||||
|
||||
```bash
|
||||
export MODEL=/path/to/MiniMax-H3/FL2VA
|
||||
CUDA_VISIBLE_DEVICES=0,1 VLLM_WORKER_MULTIPROC_METHOD=spawn \
|
||||
VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400 \
|
||||
vllm serve "$MODEL" --omni --host 127.0.0.1 --port 8000 \
|
||||
--served-model-name MiniMaxAI/MiniMax-H3 --trust-remote-code \
|
||||
--task-type fl2va --num-gpus 2 --tensor-parallel-size 2 \
|
||||
--text-encoder-tp-size 2 --enforce-eager
|
||||
```
|
||||
|
||||
Set Generate Video's **url** to the address reachable from ComfyUI, including
|
||||
`/v1`, and match its **model** to the served name. The defaults are
|
||||
`http://localhost:8000/v1` and `MiniMaxAI/MiniMax-H3`.
|
||||
|
||||
## Base and Turbo
|
||||
|
||||
Base presets are connected by default. Write the scene and its sound in the
|
||||
same **prompt**; leave **frame**, **references**, and **fast_h3** disconnected.
|
||||
The template uses 1344×768, 24 FPS, seed 1101 and **duration** 5.167 seconds
|
||||
(124 frames, satisfying H3's `17k+5` constraint). The existing client infers
|
||||
`16:9` from these dimensions. For portrait output, use 768×1344.
|
||||
|
||||
| Setting | Base | Turbo v1.0 768p |
|
||||
| --- | --- | --- |
|
||||
| Inference steps | 50 | 5 (4 forwards) |
|
||||
| Video flow shift | 12 | 6 |
|
||||
| Audio flow shift | 3 | 3 |
|
||||
| Remote LoRA | Disconnected | Connected, scale 1 |
|
||||
|
||||
For Turbo, use the **Diffusers-layout** artifact from
|
||||
[lightx2v/Minimax-h3-Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo):
|
||||
|
||||
```text
|
||||
minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors
|
||||
```
|
||||
|
||||
Add `--lora-backend peft --lora-path "$TURBO_LORA"` to the server command,
|
||||
where `TURBO_LORA` is the full server-side path to that file. Enter the same
|
||||
path in Remote LoRA's **local_path**; retain name `h3-turbo-v1.0-768p`, scale
|
||||
`1.0`, and **int_id** `0` so the existing API derives the ID from the path.
|
||||
The template leaves the path blank for users to configure. The `_comfyui_`
|
||||
export is not supported by this backend.
|
||||
|
||||
Connect **Turbo sampling**, **Turbo H3 params**, and **Remote LoRA** to Generate
|
||||
Video's `sampling_params`, `model_params`, and `lora` inputs, replacing the
|
||||
Base links. To return to Base, restore both Base links and disconnect LoRA.
|
||||
Other Turbo artifacts may need different settings; see the
|
||||
[artifact table](../../../recipes/MiniMaxAI/MiniMax-H3.md#turbo-lora).
|
||||
|
||||
If importing an older template with a frame-count widget, reimport this JSON
|
||||
or set **duration** to **5.167**; do not interpret 124 frames as 124 seconds.
|
||||
|
||||
## Validation
|
||||
|
||||
Queue the configured workflow and inspect the MP4 saved under
|
||||
`ComfyUI/output/video/MiniMax_H3_WF01_*.mp4`:
|
||||
|
||||
```bash
|
||||
ffprobe -v error -show_streams -show_format -of json "$OUTPUT"
|
||||
ffmpeg -v error -i "$OUTPUT" -f null -
|
||||
```
|
||||
|
||||
Check the requested dimensions, frame count, 24 FPS and both video/audio
|
||||
streams, then watch and listen to the saved file. The current upstream
|
||||
extension includes the shared audio-preserving decoder fix.
|
||||
|
||||
On September 16, the narrowed workflow was validated through ComfyUI using
|
||||
existing upstream Python code at `4a8ac297`, two H20-3e GPUs, vLLM 0.29.0,
|
||||
PyTorch 2.13.0+cu129 and ComfyUI `1d48d9cf`. The run used the template's
|
||||
forest-stream prompt, explicit server-side Turbo LoRA path, and settings
|
||||
above, with `TORCH_SDPA`, `--usp 1 --ring 1`, and tiled VAE parallelism on two
|
||||
GPUs. ComfyUI execution took 131.55 seconds. SaveVideo produced 124 H.264
|
||||
frames at 1344×768 and 24 FPS, plus 32 kHz stereo AAC audio. Video/audio
|
||||
durations were 5.166667/5.167000 seconds; full decoding passed and audio was
|
||||
nonzero. This is model-generated output, not a ground-truth reference or a
|
||||
subjective audio-quality assessment. Base graph validation passed with Turbo
|
||||
nodes disconnected; full 50-step Base quality was not evaluated.
|
||||
|
||||
Based on the official
|
||||
[ComfyUI H3 T2V template](https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_t2v.json),
|
||||
using remote inference instead of local model loaders.
|
||||
@@ -0,0 +1,362 @@
|
||||
{
|
||||
"last_node_id": 8,
|
||||
"last_link_id": 3,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "VLLMOmniDiffusionSampling",
|
||||
"pos": [
|
||||
20,
|
||||
20
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "diffusion sampling params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"links": [
|
||||
1
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniDiffusionSampling"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
50,
|
||||
1.0,
|
||||
1.0,
|
||||
false,
|
||||
false,
|
||||
1101,
|
||||
"fixed"
|
||||
],
|
||||
"title": "Base sampling"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [
|
||||
20,
|
||||
340
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
140
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MiniMax-H3 video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": [
|
||||
2
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniMiniMaxH3Params"
|
||||
},
|
||||
"widgets_values": [
|
||||
3.0,
|
||||
12.0
|
||||
],
|
||||
"title": "Base H3 params"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [
|
||||
430,
|
||||
20
|
||||
],
|
||||
"size": [
|
||||
430,
|
||||
440
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "frame",
|
||||
"type": "IMAGE",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "references",
|
||||
"type": "VIDEO_REFERENCES",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "sampling_params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "lora",
|
||||
"type": "REMOTE_LORA",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "model_params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"name": "fast_h3",
|
||||
"type": "FASTH3_DEPLOYMENT",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "duration",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "duration"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
3
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniGenerateVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"http://localhost:8000/v1",
|
||||
"MiniMaxAI/MiniMax-H3",
|
||||
"A small stream flows through a sunlit forest. The camera slowly moves toward the water. Birds chirp in the trees and water gurgles over smooth stones. Natural continuous sound, no music.",
|
||||
"",
|
||||
1344,
|
||||
768,
|
||||
24,
|
||||
5.167
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "SaveVideo",
|
||||
"pos": [
|
||||
940,
|
||||
20
|
||||
],
|
||||
"size": [
|
||||
440,
|
||||
440
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"link": 3
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": null,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SaveVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"video/MiniMax_H3_WF01",
|
||||
"mp4",
|
||||
"auto"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VLLMOmniRemoteLoRA",
|
||||
"pos": [
|
||||
430,
|
||||
540
|
||||
],
|
||||
"size": [
|
||||
430,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "lora",
|
||||
"type": "REMOTE_LORA",
|
||||
"links": null,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniRemoteLoRA"
|
||||
},
|
||||
"widgets_values": [
|
||||
"",
|
||||
"h3-turbo-v1.0-768p",
|
||||
1.0,
|
||||
0
|
||||
],
|
||||
"title": "Optional Turbo: server LoRA path"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "VLLMOmniDiffusionSampling",
|
||||
"pos": [
|
||||
20,
|
||||
540
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "diffusion sampling params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"links": null,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniDiffusionSampling"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
5,
|
||||
1.0,
|
||||
1.0,
|
||||
false,
|
||||
false,
|
||||
1101,
|
||||
"fixed"
|
||||
],
|
||||
"title": "Turbo sampling (4 forwards)"
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [
|
||||
430,
|
||||
740
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
140
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MiniMax-H3 video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": null,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniMiniMaxH3Params"
|
||||
},
|
||||
"widgets_values": [
|
||||
3.0,
|
||||
6.0
|
||||
],
|
||||
"title": "Turbo H3 params"
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "Note",
|
||||
"pos": [
|
||||
940,
|
||||
520
|
||||
],
|
||||
"size": [
|
||||
440,
|
||||
440
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"properties": {
|
||||
"Node name for S&R": "Note"
|
||||
},
|
||||
"widgets_values": [
|
||||
"MiniMax H3 Text to Video (WF-01)\n\nBASE: connected presets, 50 steps / video flow 12 / audio flow 3.\n\nTURBO: set Remote LoRA local_path to the SERVER path of minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors from lightx2v/Minimax-h3-Turbo. Keep name h3-turbo-v1.0-768p, scale 1.0 and int_id 0. Connect Turbo sampling, Turbo H3 params, and LoRA to Generate Video, replacing the two Base links. Use 5 steps (4 forwards), video flow 6 and audio flow 3. Do not use the ComfyUI-layout artifact.\n\nKeep frame, references and fast_h3 disconnected. 1344x768, 24 FPS, duration 5.167 seconds = 124 frames (17*7+5). The client selects the named 16:9 preset from the dimensions. Save Video preserves video and audio.\n\nSet URL and served model name for your server. See docs/minimax-h3-t2v.md for setup and validation. Source: Comfy-Org/workflow_templates/templates/video_minimax_h3_t2v.json."
|
||||
],
|
||||
"title": "Setup / Base and Turbo"
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
3,
|
||||
2,
|
||||
"SAMPLING_PARAMS"
|
||||
],
|
||||
[
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
3,
|
||||
4,
|
||||
"VIDEO_PARAMS"
|
||||
],
|
||||
[
|
||||
3,
|
||||
3,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"VIDEO"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 0.8,
|
||||
"offset": [
|
||||
50,
|
||||
50
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
Reference in new issue
Block a user