[Diffusion] Add Minimax-H3 modular pipeline support (#5720)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: David Chen <530634352@qq.com>
Co-authored-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
Co-authored-by: David Chen <530634352@qq.com>
This commit is contained in:
authored and GitHub committed 2026-08-06 11:43:23 +00:00
1 parent d219d93bbb
commit 48298030d4
25 files changed
+843 -177

No files matched your search

+13 -14
View File
@@ -12,19 +12,17 @@
- Maintainer: Community
This recipe adapts [MiniMax-H3.md](MiniMax-H3.md) for MUSA environments.
MiniMax H3 has two independently served checkpoint partitions:
- `FL2VA` serves text-to-video+audio (`t2va`) and
first-frame-to-video+audio (`fl2va`).
- `Ref2VA` serves image/audio or video-reference generation (`ref2va`).
The modular H3 pipeline can load both task-specific DiTs, but this MUSA recipe
uses a **single task partition**: `FL2VA` for `t2va`/`fl2va`, or `Ref2VA` for
`ref2va`. This avoids loading both DiTs.
## Prerequisites
### Checkpoint
Only download the partition needed by the server. Downloading the whole model
repository also retrieves duplicate shared components and the original
top-level pipeline layout.
Download the partition required by the selected startup task. The commands below
run a single-task server; do not use the modular default (combined) mode on MUSA
until its two-DiT memory and performance profile is validated.
For T2VA and FL2VA:
@@ -66,14 +64,13 @@ soundfile. Formats that libsndfile cannot read are demuxed through ffmpeg.
## Start a server
One server loads one checkpoint partition. Set `MODEL` to `FL2VA` for T2VA
and FL2VA, or to `Ref2VA` for Ref2VA.
Use the partition path and set `--task-type` explicitly: `fl2va` for T2VA/FL2VA
or `ref2va` for Ref2VA.
The validated Ref2VA configuration uses tensor parallelism across four MTT
S5000 GPUs and offloads inactive model components to CPU:
The validated Ref2VA configuration uses tensor parallelism across four MTT S5000 GPUs and offloads inactive model components to CPU:
```bash
export MODEL="${MODEL_ROOT}/FL2VA"
export MODEL="${MODEL_ROOT}/Ref2VA"
export PORT=8091
MUSA_VISIBLE_DEVICES=0,1,2,3 \
@@ -85,6 +82,7 @@ vllm serve "${MODEL}" \
--port "${PORT}" \
--trust-remote-code \
--num-gpus 4 \
--task-type ref2va \
--tensor-parallel-size 4 \
--vae-patch-parallel-size 4 \
--enable-cpu-offload \
@@ -133,7 +131,8 @@ Keep `--vae-use-tiling` enabled for this serving profile.
- Keep Ring Attention at degree 1. The current Ring path does not preserve
MiniMax H3 packed padding boundaries.
- H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.
- Each server process loads only one checkpoint partition.
- This MUSA recipe uses an explicitly selected single task partition; combined
serving loads both DiTs.
- H3 currently executes one generation request per diffusion batch.
- FP8 quantization has not been enabled for MiniMax H3.
- MP3, M4A, MP4, and reference-video audio fallback requires `ffmpeg` on
+15 -20
View File
@@ -26,12 +26,12 @@ environments. Differences from the GPU path:
### Checkpoint
Same as the GPU recipe — Hugging Face access approval is required:
Same as the GPU recipe — Hugging Face access approval is required. Authenticate
once; `vllm serve` downloads the required nested components automatically:
```bash
hf auth login
export MODEL_ROOT=/path/to/MiniMax-H3
hf download MiniMaxAI/MiniMax-H3 --local-dir "${MODEL_ROOT}"
export MODEL=MiniMaxAI/MiniMax-H3
```
### Environment
@@ -73,24 +73,21 @@ pip install mindiesd-*.whl
## Start a server
One server loads one checkpoint partition. Set `MODEL` to `FL2VA` for T2VA
and FL2VA requests, or to `Ref2VA` for Ref2VA requests.
Pass the repository ID directly. The pipeline loads the two nested DiTs while
sharing the tokenizer, processor, text encoder, and VAEs from `FL2VA`.
### Multi-NPU: 768P validated configuration
### Multi-NPU: 768P combined-service configuration
Validated on eight NPUs of an Atlas 800I A3 server with Ulysses sequence
parallelism degree 8, text-encoder tensor parallelism degree 8, native tiled
VAE patch parallelism degree 8, and layerwise offload:
Use Ulysses sequence parallelism degree 8, text-encoder tensor parallelism
degree 8, native tiled VAE patch parallelism degree 8, and layerwise offload:
```bash
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PORT=9098
export MODEL="${MODEL_ROOT}/FL2VA"
export MODEL=MiniMaxAI/MiniMax-H3
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800
export PYTHONDONTWRITEBYTECODE=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
vllm serve "${MODEL}" \
--omni \
@@ -112,6 +109,9 @@ Do not add `--enforce-eager`. The first request includes regional
compilation; warm the server once before measuring steady-state latency.
H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.
The same endpoint accepts `task=t2va`, `task=fl2va`, and `task=ref2va`; no
partition restart is required. Layerwise offload applies to both DiTs.
### Optional optimizations
Two independent optimizations may be enabled on top of the configuration
@@ -124,7 +124,7 @@ every other flag unchanged:
--diffusion-attention-backend RAINFUSION_ATTN
```
**INT8 online quantization** — add one flag to either command above:
**INT8 online quantization** — add one flag to the server command above:
```bash
--quantization int8
@@ -134,12 +134,6 @@ Keep `--ring 1` when using RainFusion: the `rf_v2` kernel ranks key blocks
over the whole sequence, so ring parallelism would split away the keys it
needs. Scale with `--usp` instead.
To serve Ref2VA, stop the FL2VA server and restart with:
```bash
export MODEL="${MODEL_ROOT}/Ref2VA"
```
## HTTP API examples
The request format is identical to the GPU recipe; see
@@ -168,7 +162,8 @@ throughput guarantee.
## Known limitations
- Each server process loads only one checkpoint partition.
- Combined serving requires sibling `FL2VA` and `Ref2VA` directories, loads
both task-specific DiTs, and loads one copy of every shared component.
- H3 currently executes one generation request per diffusion batch.
- The first regional-compile request is a warmup and should not be included
in steady-state performance measurements.
+45 -34
View File
@@ -12,27 +12,34 @@
- Maintainer: Community
MiniMax H3 is a CFG-distilled joint video/audio diffusion transformer. Its
checkpoint has two independently served partitions:
checkpoint has two task-specific DiT partitions:
- `FL2VA`: text-to-video+audio (`t2va`) and first-frame-to-video+audio
(`fl2va`)
- `Ref2VA`: up to 9 images, 3 videos, and 3 audio references in supported
image/video/audio combinations (`ref2va`; audio-only is rejected)
One vLLM-Omni diffusion stage can load both DiTs while instantiating the
tokenizer, processor, Qwen3-VL text encoder, video VAE, and audio VAE only
once. Requests select the DiT with `extra_params.task`.
The generated MP4 contains H.264 video and synchronized stereo audio.
## Prerequisites
The checkpoint requires Hugging Face access approval. Download it and point
`MODEL_ROOT` at the directory containing the `FL2VA` and `Ref2VA`
subdirectories:
The checkpoint requires Hugging Face access approval. Authenticate once;
`vllm serve` downloads the required components automatically:
```bash
hf auth login
export MODEL_ROOT=/path/to/MiniMax-H3
hf download MiniMaxAI/MiniMax-H3 --local-dir "${MODEL_ROOT}"
export MODEL=MiniMaxAI/MiniMax-H3
```
The vLLM-Omni pipeline downloads `FL2VA/**`, `Ref2VA/model_index.json`, and
`Ref2VA/transformer/**`. It does not download or load the diffusers-format
`transformer`, `transformer_ref`, or `vae` weights at the repository root, nor
duplicate Ref2VA copies of shared components.
Install vLLM-Omni from the checkout containing MiniMax H3 support. The
two-GPU RTX 5090/4090 profiles use cuDNN attention and do not need
FlashAttention-4. Install the optional dependency only for the four-GPU
@@ -56,16 +63,17 @@ reference-video preparation and MP4 output.
## Start a server
One server loads one checkpoint partition. Set `MODEL` to `FL2VA` for T2VA
and FL2VA requests, or to `Ref2VA` for either Ref2VA request.
Pass the repository ID directly. The pipeline uses `FL2VA` for model discovery
and shared components, and loads the second DiT from `Ref2VA/transformer`.
### Memory and storage requirements
Treat GPU HBM, host RAM, and checkpoint storage as separate requirements. Each
H3 checkpoint partition (`FL2VA` or `Ref2VA`) contains about **134 GiB** of
BF16 safetensors (about **135 GiB** on disk). Keeping both partitions locally
therefore needs roughly **270 GiB** of model storage, although the examples
serve only one partition at a time.
BF16 safetensors (about **135 GiB** on disk). Keeping both partitions
locally therefore needs roughly **270 GiB** of model storage. A combined
service downloads both; `--task-type fl2va` or `--task-type ref2va` downloads
only the selected partition.
CPU offload and distributed layerwise offload reduce GPU residency; they do
not make the model weights disappear. With `--dlo-no-use-allgather`, each
@@ -86,7 +94,7 @@ This matches the accuracy-qualified reference path and prevents the Qwen3-VL
encoder and DiT from being resident on the GPU at the same time.
```bash
export MODEL="${MODEL_ROOT}/FL2VA"
export MODEL=MiniMaxAI/MiniMax-H3
export PORT=8091
CUDA_VISIBLE_DEVICES=0 \
@@ -103,8 +111,8 @@ vllm serve "${MODEL}" \
```
Use a GPU with enough memory for the active H3 component and enough system RAM
for the offloaded components. CPU offload reduces GPU memory pressure but adds
PCIe/NVLink transfer latency.
for both offloaded DiTs plus the shared components. Model-level offload keeps
the two DiTs mutually exclusive on GPU, but adds PCIe/NVLink transfer latency.
### Two 24/32 GB GPUs: TP2 distributed layerwise offload
@@ -116,7 +124,7 @@ copied to the GPUs once per denoise stage, reused by every sampling step, and
released before VAE decode so the decoder can reuse their HBM.
```bash
export MODEL="${MODEL_ROOT}/FL2VA"
export MODEL=MiniMaxAI/MiniMax-H3
export PORT=8091
CUDA_VISIBLE_DEVICES=0,1 \
@@ -127,6 +135,7 @@ vllm serve "${MODEL}" \
--host 0.0.0.0 \
--port "${PORT}" \
--trust-remote-code \
--task-type fl2va \
--num-gpus 2 \
--tensor-parallel-size 2 \
--usp 1 \
@@ -198,9 +207,9 @@ bash examples/offline_inference/minimax_h3/run_h3_2gpu_all_tasks.sh
The script selects 20 resident layers for `PROFILE=rtx5090` and 12 for
`PROFILE=rtx4090`; `DLO_RESIDENT_LAYERS=N` overrides either default.
### Four GPUs: measured best-practice throughput
### Four GPUs: throughput-oriented combined service
The best validated four-GPU configuration on four NVIDIA B300 GPUs is:
For a combined service on four high-memory GPUs, use:
- no CPU or layerwise offload;
- Ulysses sequence parallelism degree 4;
@@ -208,8 +217,11 @@ The best validated four-GPU configuration on four NVIDIA B300 GPUs is:
- regional `torch.compile` for the repeated DiT blocks;
- dense BF16 `TRTLLM_ATTN`, with Ring and TP left at 1.
Both DiTs remain resident in this no-offload configuration. If they do not fit,
use model-level CPU offload.
```bash
export MODEL="${MODEL_ROOT}/FL2VA"
export MODEL=MiniMaxAI/MiniMax-H3
export PORT=8091
CUDA_VISIBLE_DEVICES=0,1,2,3 \
@@ -305,7 +317,7 @@ vLLM-style tensor-parallel layers and runs with distributed collectives over
its own encoder process group):
```bash
export MODEL="${MODEL_ROOT}/FL2VA"
export MODEL=MiniMaxAI/MiniMax-H3
export PORT=8091
CUDA_VISIBLE_DEVICES=0,1,2,3 \
@@ -335,12 +347,9 @@ the reference path within bf16 rounding: every encoder rank all-reduces the
row-parallel projections, so the full `[seq, 5120]` layer-50 hidden state is
replicated on every rank.
To serve Ref2VA, stop the FL2VA server and restart the same one- or four-GPU
command with:
```bash
export MODEL="${MODEL_ROOT}/Ref2VA"
```
No restart is needed: `task=fl2va` routes to `FL2VA/transformer`, while
`task=ref2va` routes to
`Ref2VA/transformer`. T2VA uses the FL2VA DiT.
### Online FP8 quantization
@@ -396,7 +405,7 @@ export API_URL="http://127.0.0.1:${PORT}/v1/videos/sync"
### 1. T2VA: text to video and audio
Run this request against the `FL2VA` partition:
Run this request against the combined service:
```bash
curl -sS -X POST "${API_URL}" \
@@ -414,7 +423,7 @@ curl -sS -X POST "${API_URL}" \
### 2. FL2VA: first frame to video and audio
Run this request against the `FL2VA` partition. When width and height are
Run this request against the combined service. When width and height are
omitted, H3 preserves the first-frame aspect ratio and uses a 768-pixel short
edge.
@@ -453,10 +462,11 @@ curl -sS -X POST "${API_URL}" \
### 3. Ref2VA: image-only, image/audio, or mixed references
Run these requests against the `Ref2VA` partition. Image-only Ref2VA omits
`audio_reference`; adding one or more audio references is optional. The typed
fields accept one object or an ordered JSON list. `audio_reference` accepts an
HTTP(S) URL or a `data:` URL.
Run these requests against the combined service or a Ref2VA-only service.
Image-only Ref2VA omits `audio_reference`; adding one or more audio references
is optional. The typed fields accept one object or an ordered JSON list.
`audio_reference` accepts an HTTP(S) URL or a `data:` URL. In one terminal,
expose the local reference assets to the serving host:
```bash
python -m http.server 8092 \
@@ -506,7 +516,7 @@ shorter, the reference soundtrack is truncated to the generated clip.
### 4. Ref2VA: video, separate audio, and mixed references
Run this request against the `Ref2VA` partition. Repeat the
Run this request against the combined service. Repeat the
`input_references` multipart field once per source video. H3 consumes the
videos in form order and preserves their original soundtracks during
conditioning.
@@ -568,7 +578,7 @@ than one is requested.
| Parameter | Recommended value | Notes |
|-----------|-------------------|-------|
| `task` | `t2va`, `fl2va`, or `ref2va` | Passed in `extra_params`; must match the served partition |
| `task` | `t2va`, `fl2va`, or `ref2va` | Passed in `extra_params`; selects the task-specific DiT |
| `duration` | Workload-specific | Decimal seconds in `extra_params`; converted to H3-compatible frame count |
| `fps` | `24` | H3 output FPS is fixed |
| `num_inference_steps` | `50` | Matches the reference accuracy workloads |
@@ -610,7 +620,8 @@ reduction.
## Known limitations
- Each server process loads only one checkpoint partition.
- Combined serving requires sibling `FL2VA` and `Ref2VA` directories, loads
both task-specific DiTs, and loads shared components once from `FL2VA`.
- H3 currently executes one generation request per diffusion batch.
- The first regional-compile request is a warmup and should not be included in
steady-state performance measurements.