diff --git a/apps/ComfyUI-vLLM-Omni/README.md b/apps/ComfyUI-vLLM-Omni/README.md index 8f445635d..d0d002d0f 100644 --- a/apps/ComfyUI-vLLM-Omni/README.md +++ b/apps/ComfyUI-vLLM-Omni/README.md @@ -213,6 +213,15 @@ Record the server commit, model/adapter, input assets, prompt, settings, and out alongside the result. Local schema or mocked-server checks do not establish H3 generation quality or replace this real-model validation. +### MiniMax H3 Text to Video + +Import [the H3 text-to-video template](example_workflows/MiniMax_H3_Text_to_Video.json) +for remote video generation with audio. It includes connected base presets and +optional Turbo sampling, H3 parameters, and Remote LoRA nodes. See the +[workflow guide](docs/minimax-h3-t2v.md) for server setup, artifact-specific Turbo +settings, and saved-video/audio validation. Set Remote LoRA’s `local_path` to +the downloaded Turbo artifact on your server before enabling Turbo. + #### H3 video upscale (WF-07) The **vLLM-Omni MiniMax H3 Video Upscale** template generates video remotely, upscales it with SeedVR2, and saves the original and upscaled videos with the generated audio and FPS. See [workflow setup](docs/wf07-h3-upscale.md). diff --git a/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md b/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md new file mode 100644 index 000000000..790aa0f53 --- /dev/null +++ b/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md @@ -0,0 +1,94 @@ +# MiniMax H3 Text to Video (WF-01) + +Import [MiniMax_H3_Text_to_Video.json](../example_workflows/MiniMax_H3_Text_to_Video.json) +into ComfyUI, or select it under **Templates → ComfyUI-vLLM-Omni**. +The template connects existing nodes to a remote H3 service and saves the +returned video with audio. ComfyUI does not load H3 weights. + +## Setup + +Install the [ComfyUI extension](../README.md#installation) and follow the +[H3 deployment recipe](../../../recipes/MiniMaxAI/MiniMax-H3.md) for your hardware. +For a two-GPU FL2VA server with sufficient memory: + +```bash +export MODEL=/path/to/MiniMax-H3/FL2VA +CUDA_VISIBLE_DEVICES=0,1 VLLM_WORKER_MULTIPROC_METHOD=spawn \ +VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400 \ +vllm serve "$MODEL" --omni --host 127.0.0.1 --port 8000 \ + --served-model-name MiniMaxAI/MiniMax-H3 --trust-remote-code \ + --task-type fl2va --num-gpus 2 --tensor-parallel-size 2 \ + --text-encoder-tp-size 2 --enforce-eager +``` + +Set Generate Video's **url** to the address reachable from ComfyUI, including +`/v1`, and match its **model** to the served name. The defaults are +`http://localhost:8000/v1` and `MiniMaxAI/MiniMax-H3`. + +## Base and Turbo + +Base presets are connected by default. Write the scene and its sound in the +same **prompt**; leave **frame**, **references**, and **fast_h3** disconnected. +The template uses 1344×768, 24 FPS, seed 1101 and **duration** 5.167 seconds +(124 frames, satisfying H3's `17k+5` constraint). The existing client infers +`16:9` from these dimensions. For portrait output, use 768×1344. + +| Setting | Base | Turbo v1.0 768p | +| --- | --- | --- | +| Inference steps | 50 | 5 (4 forwards) | +| Video flow shift | 12 | 6 | +| Audio flow shift | 3 | 3 | +| Remote LoRA | Disconnected | Connected, scale 1 | + +For Turbo, use the **Diffusers-layout** artifact from +[lightx2v/Minimax-h3-Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo): + +```text +minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors +``` + +Add `--lora-backend peft --lora-path "$TURBO_LORA"` to the server command, +where `TURBO_LORA` is the full server-side path to that file. Enter the same +path in Remote LoRA's **local_path**; retain name `h3-turbo-v1.0-768p`, scale +`1.0`, and **int_id** `0` so the existing API derives the ID from the path. +The template leaves the path blank for users to configure. The `_comfyui_` +export is not supported by this backend. + +Connect **Turbo sampling**, **Turbo H3 params**, and **Remote LoRA** to Generate +Video's `sampling_params`, `model_params`, and `lora` inputs, replacing the +Base links. To return to Base, restore both Base links and disconnect LoRA. +Other Turbo artifacts may need different settings; see the +[artifact table](../../../recipes/MiniMaxAI/MiniMax-H3.md#turbo-lora). + +If importing an older template with a frame-count widget, reimport this JSON +or set **duration** to **5.167**; do not interpret 124 frames as 124 seconds. + +## Validation + +Queue the configured workflow and inspect the MP4 saved under +`ComfyUI/output/video/MiniMax_H3_WF01_*.mp4`: + +```bash +ffprobe -v error -show_streams -show_format -of json "$OUTPUT" +ffmpeg -v error -i "$OUTPUT" -f null - +``` + +Check the requested dimensions, frame count, 24 FPS and both video/audio +streams, then watch and listen to the saved file. The current upstream +extension includes the shared audio-preserving decoder fix. + +On September 16, the narrowed workflow was validated through ComfyUI using +existing upstream Python code at `4a8ac297`, two H20-3e GPUs, vLLM 0.29.0, +PyTorch 2.13.0+cu129 and ComfyUI `1d48d9cf`. The run used the template's +forest-stream prompt, explicit server-side Turbo LoRA path, and settings +above, with `TORCH_SDPA`, `--usp 1 --ring 1`, and tiled VAE parallelism on two +GPUs. ComfyUI execution took 131.55 seconds. SaveVideo produced 124 H.264 +frames at 1344×768 and 24 FPS, plus 32 kHz stereo AAC audio. Video/audio +durations were 5.166667/5.167000 seconds; full decoding passed and audio was +nonzero. This is model-generated output, not a ground-truth reference or a +subjective audio-quality assessment. Base graph validation passed with Turbo +nodes disconnected; full 50-step Base quality was not evaluated. + +Based on the official +[ComfyUI H3 T2V template](https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_t2v.json), +using remote inference instead of local model loaders. diff --git a/apps/ComfyUI-vLLM-Omni/example_workflows/MiniMax_H3_Text_to_Video.json b/apps/ComfyUI-vLLM-Omni/example_workflows/MiniMax_H3_Text_to_Video.json new file mode 100644 index 000000000..137ee8c2d --- /dev/null +++ b/apps/ComfyUI-vLLM-Omni/example_workflows/MiniMax_H3_Text_to_Video.json @@ -0,0 +1,362 @@ +{ + "last_node_id": 8, + "last_link_id": 3, + "nodes": [ + { + "id": 1, + "type": "VLLMOmniDiffusionSampling", + "pos": [ + 20, + 20 + ], + "size": [ + 310, + 280 + ], + "flags": {}, + "order": 0, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "diffusion sampling params", + "type": "SAMPLING_PARAMS", + "links": [ + 1 + ], + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VLLMOmniDiffusionSampling" + }, + "widgets_values": [ + 1, + 50, + 1.0, + 1.0, + false, + false, + 1101, + "fixed" + ], + "title": "Base sampling" + }, + { + "id": 2, + "type": "VLLMOmniMiniMaxH3Params", + "pos": [ + 20, + 340 + ], + "size": [ + 310, + 140 + ], + "flags": {}, + "order": 1, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "MiniMax-H3 video params", + "type": "VIDEO_PARAMS", + "links": [ + 2 + ], + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VLLMOmniMiniMaxH3Params" + }, + "widgets_values": [ + 3.0, + 12.0 + ], + "title": "Base H3 params" + }, + { + "id": 3, + "type": "VLLMOmniGenerateVideo", + "pos": [ + 430, + 20 + ], + "size": [ + 430, + 440 + ], + "flags": {}, + "order": 2, + "mode": 0, + "inputs": [ + { + "name": "frame", + "type": "IMAGE", + "link": null + }, + { + "name": "references", + "type": "VIDEO_REFERENCES", + "link": null + }, + { + "name": "sampling_params", + "type": "SAMPLING_PARAMS", + "link": 1 + }, + { + "name": "lora", + "type": "REMOTE_LORA", + "link": null + }, + { + "name": "model_params", + "type": "VIDEO_PARAMS", + "link": 2 + }, + { + "name": "fast_h3", + "type": "FASTH3_DEPLOYMENT", + "link": null + }, + { + "name": "duration", + "type": "FLOAT", + "widget": { + "name": "duration" + }, + "link": null + } + ], + "outputs": [ + { + "name": "video", + "type": "VIDEO", + "links": [ + 3 + ], + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VLLMOmniGenerateVideo" + }, + "widgets_values": [ + "http://localhost:8000/v1", + "MiniMaxAI/MiniMax-H3", + "A small stream flows through a sunlit forest. The camera slowly moves toward the water. Birds chirp in the trees and water gurgles over smooth stones. Natural continuous sound, no music.", + "", + 1344, + 768, + 24, + 5.167 + ] + }, + { + "id": 4, + "type": "SaveVideo", + "pos": [ + 940, + 20 + ], + "size": [ + 440, + 440 + ], + "flags": {}, + "order": 3, + "mode": 0, + "inputs": [ + { + "name": "video", + "type": "VIDEO", + "link": 3 + } + ], + "outputs": [ + { + "name": "video", + "type": "VIDEO", + "links": null, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "SaveVideo" + }, + "widgets_values": [ + "video/MiniMax_H3_WF01", + "mp4", + "auto" + ] + }, + { + "id": 5, + "type": "VLLMOmniRemoteLoRA", + "pos": [ + 430, + 540 + ], + "size": [ + 430, + 160 + ], + "flags": {}, + "order": 4, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "lora", + "type": "REMOTE_LORA", + "links": null, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VLLMOmniRemoteLoRA" + }, + "widgets_values": [ + "", + "h3-turbo-v1.0-768p", + 1.0, + 0 + ], + "title": "Optional Turbo: server LoRA path" + }, + { + "id": 6, + "type": "VLLMOmniDiffusionSampling", + "pos": [ + 20, + 540 + ], + "size": [ + 310, + 280 + ], + "flags": {}, + "order": 5, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "diffusion sampling params", + "type": "SAMPLING_PARAMS", + "links": null, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VLLMOmniDiffusionSampling" + }, + "widgets_values": [ + 1, + 5, + 1.0, + 1.0, + false, + false, + 1101, + "fixed" + ], + "title": "Turbo sampling (4 forwards)" + }, + { + "id": 7, + "type": "VLLMOmniMiniMaxH3Params", + "pos": [ + 430, + 740 + ], + "size": [ + 310, + 140 + ], + "flags": {}, + "order": 6, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "MiniMax-H3 video params", + "type": "VIDEO_PARAMS", + "links": null, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VLLMOmniMiniMaxH3Params" + }, + "widgets_values": [ + 3.0, + 6.0 + ], + "title": "Turbo H3 params" + }, + { + "id": 8, + "type": "Note", + "pos": [ + 940, + 520 + ], + "size": [ + 440, + 440 + ], + "flags": {}, + "order": 7, + "mode": 0, + "inputs": [], + "outputs": [], + "properties": { + "Node name for S&R": "Note" + }, + "widgets_values": [ + "MiniMax H3 Text to Video (WF-01)\n\nBASE: connected presets, 50 steps / video flow 12 / audio flow 3.\n\nTURBO: set Remote LoRA local_path to the SERVER path of minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors from lightx2v/Minimax-h3-Turbo. Keep name h3-turbo-v1.0-768p, scale 1.0 and int_id 0. Connect Turbo sampling, Turbo H3 params, and LoRA to Generate Video, replacing the two Base links. Use 5 steps (4 forwards), video flow 6 and audio flow 3. Do not use the ComfyUI-layout artifact.\n\nKeep frame, references and fast_h3 disconnected. 1344x768, 24 FPS, duration 5.167 seconds = 124 frames (17*7+5). The client selects the named 16:9 preset from the dimensions. Save Video preserves video and audio.\n\nSet URL and served model name for your server. See docs/minimax-h3-t2v.md for setup and validation. Source: Comfy-Org/workflow_templates/templates/video_minimax_h3_t2v.json." + ], + "title": "Setup / Base and Turbo" + } + ], + "links": [ + [ + 1, + 1, + 0, + 3, + 2, + "SAMPLING_PARAMS" + ], + [ + 2, + 2, + 0, + 3, + 4, + "VIDEO_PARAMS" + ], + [ + 3, + 3, + 0, + 4, + 0, + "VIDEO" + ] + ], + "groups": [], + "config": {}, + "extra": { + "ds": { + "scale": 0.8, + "offset": [ + 50, + 50 + ] + } + }, + "version": 0.4 +} diff --git a/docs/features/comfyui.md b/docs/features/comfyui.md index b4d6fa101..876e80617 100644 --- a/docs/features/comfyui.md +++ b/docs/features/comfyui.md @@ -81,6 +81,18 @@ To build a simple workflow yourself, - For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters. - For multi-stage models where all stages are either autoregression or diffusion, you can also connect only a single Sampling Params node, indicating that this set of sampling parameters will be used for all stages. +## MiniMax H3 text-to-video workflow + +The **MiniMax H3 Text to Video** template uses the existing remote Generate Video, +H3 Params, sampling, and Remote LoRA nodes. It includes Base settings and an +optional Turbo preset, with native resolution, 24 FPS, and `17k+5` frame counts. +ComfyUI receives the generated video and audio from the server; it does not load +H3 weights locally. Configure the server-side LoRA path before enabling Turbo. + +See the [H3 workflow guide](https://github.com/vllm-project/vllm-omni/blob/main/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md) +for server setup, template import, Turbo configuration, recorded validation, +and saved-video/audio checks. + ## Examples & Screenshots Please read the [ComfyUI integration's README](https://github.com/vllm-project/vllm-omni/tree/main/apps/ComfyUI-vLLM-Omni) for more info.