[Model] Add native MAGI-2 Preview diffusion support (#5918)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu authored and GitHub committed
1 parent
13c617a140
commit
d4096e01fe
78 files changed
+9896
-459
No files matched your search
@@ -0,0 +1,288 @@
|
||||
# MAGI-2 Preview — NVIDIA L20X
|
||||
|
||||
> Native text/image-to-video-and-audio generation with the released Preview stage
|
||||
|
||||
## Summary
|
||||
|
||||
- Vendor: SandAI
|
||||
- Model: `sand-ai/MAGI-2-preview`
|
||||
- Runtime: Native vLLM-Omni pipeline; no SandAI runtime import
|
||||
- Modes: Offline shared examples and OpenAI-compatible video serving
|
||||
- Hardware: NVIDIA L20X 140 GiB
|
||||
- Recommended deployment: Four-GPU resident SP4 (`TP=1`, `SP=4`)
|
||||
- Maintainer: Community
|
||||
|
||||
## Supported model contract
|
||||
|
||||
### Tasks and inputs
|
||||
|
||||
| Task | Text prompt | Media condition | Shared entrypoint |
|
||||
| --- | --- | --- | --- |
|
||||
| T2VA | Required | None | [`text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py) |
|
||||
| I2VA | Required | Exactly one still image | [`image_to_video.py`](../../examples/offline_inference/image_to_video/image_to_video.py) |
|
||||
|
||||
The I2VA path preserves the source aspect ratio and applies MAGI-2's native
|
||||
resize/pad transform. Video or audio conditions, multiple images, and fused
|
||||
per-rank request batches are not supported.
|
||||
|
||||
### Output
|
||||
|
||||
| Category | Supported Preview specification |
|
||||
| --- | --- |
|
||||
| Duration | Exactly 10 seconds |
|
||||
| Video | 125 frames at 12.5 fps |
|
||||
| Native tiers | `272p` (448x256) and `540p` (896x512) |
|
||||
| Optional final resize | `output_width` and `output_height` must be supplied together |
|
||||
| Audio | 44.1 kHz stereo, generated with the video |
|
||||
| Refiner | The separate 1080p refiner is not included in this integration |
|
||||
|
||||
### Profiles provided by this recipe
|
||||
|
||||
| Profile | Devices | Purpose | Qualification |
|
||||
| --- | ---: | --- | --- |
|
||||
| Resident SP4 | 4 | Recommended fidelity/default deployment | Released smoke and reference parity |
|
||||
| Rank-local DLO SP4 | 4 | Full-quality lower-HBM transformer streaming | Released checkpoint, 100-step T2VA and I2VA |
|
||||
| DLO DP4 / DP2SP2 | 4 | Concurrent request throughput | Bounded one- and four-step coverage |
|
||||
| Ordinary layerwise | 1 | Memory-constrained single request | Released checkpoint, one and four steps |
|
||||
| Online resident SP4 | 4 | OpenAI-compatible video API | Serving path covered |
|
||||
|
||||
## References
|
||||
|
||||
- Official model card: <https://huggingface.co/sand-ai/MAGI-2-preview>
|
||||
- Architecture blog: <https://sand.ai/blog/magi-2-preview>
|
||||
- Supported models: [`docs/models/supported_models.md`](../../docs/models/supported_models.md)
|
||||
- Diffusion feature matrix: [`docs/user_guide/diffusion_features.md`](../../docs/user_guide/diffusion_features.md)
|
||||
|
||||
## Prepare the checkpoint
|
||||
|
||||
The native Preview path uses approximately 274 GiB on disk. The separate
|
||||
refiner is not required.
|
||||
|
||||
```bash
|
||||
export MAGI2_CKPT_ROOT=/path/to/MAGI-2-preview
|
||||
|
||||
hf download sand-ai/MAGI-2-preview \
|
||||
preview/ text_encoder/ vae/ turbo_vae/ stable-audio-open-1.0/ \
|
||||
--revision 2dea51b64db47ee5b4402d36fd90829a0c58913b \
|
||||
--local-dir "$MAGI2_CKPT_ROOT"
|
||||
```
|
||||
|
||||
Install vLLM-Omni and ensure `ffmpeg` is on `PATH`. No second model runtime is
|
||||
required.
|
||||
|
||||
## Hardware support
|
||||
|
||||
### Locally qualified NVIDIA configuration
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| Accelerator | NVIDIA L20X |
|
||||
| Per-device memory | 143,771 MiB (about 140.4 GiB) |
|
||||
| Device count | 4; the one-device profile uses one of the same GPUs |
|
||||
| Device interconnect | All-to-all NV18 links (`nvidia-smi topo -m`) |
|
||||
| Host memory | 2.0 TiB |
|
||||
|
||||
The official SandAI runtime requires eight NVIDIA Hopper GPUs. That official
|
||||
eight-GPU system was unavailable for this PR; compatible eight-worker
|
||||
topologies pass configuration validation but are not locally runtime-qualified.
|
||||
No H100, H200, B200, or B300 runtime claim is made here.
|
||||
|
||||
### Software environment
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| OS | Ubuntu 22.04.5 LTS, Linux 5.10.134 |
|
||||
| Python | 3.12.13 |
|
||||
| NVIDIA driver / CUDA runtime | 570.133.20 / 13.0 |
|
||||
| PyTorch | 2.11.0+cu130 |
|
||||
| vLLM | 0.26.0 |
|
||||
| vLLM-Omni | This PR branch; evidence commits `a5af2a8c` and `42bbfb57` |
|
||||
| Precision | BF16 |
|
||||
|
||||
Set `MAGI2_DETERMINISTIC=1` before worker startup when deterministic kernels
|
||||
are required. The setting is fixed for the worker lifetime.
|
||||
|
||||
Each worker process supports one MAGI-2 pipeline instance. Pipeline startup
|
||||
sets process-wide deterministic state, so multi-pipeline tests and deployments
|
||||
must use separate worker processes rather than constructing multiple instances
|
||||
inside one process.
|
||||
|
||||
MAGI-2 uses vLLM's bundled FlashAttention 3 kernels by default on Hopper
|
||||
(compute capability 9.x), FlashAttention 4 on supported Blackwell devices
|
||||
with FA3 as the fallback, and FlashAttention 2 on earlier architectures. Set
|
||||
`MAGI2_FLASH_ATTN_VERSION=2`, `3`, or `4` before worker startup to force an
|
||||
available kernel for diagnostics or output comparisons.
|
||||
|
||||
## Offline generation
|
||||
|
||||
### Recommended four-GPU T2VA
|
||||
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model "$MAGI2_CKPT_ROOT" \
|
||||
--model-class-name Magi2Pipeline \
|
||||
--prompt "A red fox walks through fresh snow while wind moves the pine branches." \
|
||||
--height 512 --width 896 --num-frames 125 \
|
||||
--num-inference-steps 100 --fps 12.5 \
|
||||
--tensor-parallel-size 1 --ulysses-degree 4 \
|
||||
--extra-body '{"seconds":10,"resolution":"540p"}' \
|
||||
--output magi2_540p_t2va.mp4
|
||||
```
|
||||
|
||||
For a load-and-kernel smoke, use one inference step. A one-step output is not a
|
||||
quality evaluation.
|
||||
|
||||
### Recommended four-GPU I2VA
|
||||
|
||||
Use the same geometry and parallelism with the shared I2V entrypoint:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||||
--model "$MAGI2_CKPT_ROOT" \
|
||||
--model-class-name Magi2Pipeline \
|
||||
--image /path/to/first_frame.png \
|
||||
--prompt "The fox looks up, then walks forward as snow falls around it." \
|
||||
--height 512 --width 896 --num-frames 125 \
|
||||
--num-inference-steps 100 --fps 12.5 \
|
||||
--tensor-parallel-size 1 --ulysses-degree 4 \
|
||||
--extra-body '{"seconds":10,"resolution":"540p"}' \
|
||||
--output magi2_540p_i2va.mp4
|
||||
```
|
||||
|
||||
### One-device layerwise offload
|
||||
|
||||
MAGI-2 already stages Qwen, the image VAE, TurboVAE, and Oobleck from pinned
|
||||
CPU memory. Ordinary layerwise offload streams the Preview DiT blocks so the
|
||||
complete transformer does not need to fit in HBM at once.
|
||||
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model "$MAGI2_CKPT_ROOT" \
|
||||
--model-class-name Magi2Pipeline \
|
||||
--prompt "A red fox walks through fresh snow while wind moves the pine branches." \
|
||||
--height 256 --width 448 --num-frames 125 \
|
||||
--num-inference-steps 4 --fps 12.5 \
|
||||
--enable-cpu-offload --enable-layerwise-offload \
|
||||
--extra-body '{"seconds":10,"resolution":"272p"}' \
|
||||
--output magi2_272p_cpu_layerwise.mp4
|
||||
```
|
||||
|
||||
Standalone `--enable-cpu-offload` is rejected because staging whole modules
|
||||
does not make the complete Preview DiT fit on one qualified GPU. When both
|
||||
flags are supplied, the shared offloader selects layerwise mode and MAGI's
|
||||
native auxiliary staging remains active.
|
||||
|
||||
## Online serving
|
||||
|
||||
The OpenAI-compatible serving defaults are the full `540p` preset and `100`
|
||||
denoising steps. The model always generates a 10-second, 125-frame clip at
|
||||
12.5 fps; pass `resolution=272p` and `num_inference_steps=1` for a low-cost
|
||||
load smoke. Requests with another duration are rejected before generation.
|
||||
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
|
||||
vllm serve "$MAGI2_CKPT_ROOT" --omni \
|
||||
--model-class-name Magi2Pipeline \
|
||||
--num-gpus 4 \
|
||||
--tensor-parallel-size 1 \
|
||||
--ulysses-degree 4 \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/videos/sync \
|
||||
-F 'prompt=A red fox walks through fresh snow.' \
|
||||
-F 'seconds=10' \
|
||||
-F 'size=896x512' \
|
||||
-F 'num_frames=125' \
|
||||
-F 'fps=12.5' \
|
||||
-F 'num_inference_steps=100' \
|
||||
-F 'seed=42' \
|
||||
-F 'extra_params={"resolution":"540p"}' \
|
||||
-o magi2_online.mp4
|
||||
```
|
||||
|
||||
For I2VA, add `-F 'input_reference=@first_frame.png;type=image/png'`.
|
||||
|
||||
## Supported features
|
||||
|
||||
Use the shared guides for launch syntax and feature semantics. The flags below
|
||||
are the MAGI-2-specific qualified combinations.
|
||||
|
||||
| Feature | MAGI-2 status / topology | Guide |
|
||||
| --- | --- | --- |
|
||||
| Sequence parallel | Resident SP4 default; compatible SP8 configuration | [Sequence parallel](../../docs/user_guide/diffusion/parallelism/sequence_parallel.md) |
|
||||
| Tensor parallel | TP4 or TP2SP2 on four workers | [Tensor parallel](../../docs/user_guide/diffusion/parallelism/tensor_parallel.md) |
|
||||
| DLO | DP4/DP2SP2 AllGather; SP4 rank-local requires `--dlo-no-use-allgather` | [Distributed layerwise offload](../../docs/design/feature/distributed_layerwise_offload.md) |
|
||||
| HSDP | HSDP4+SP4; alternative to TP and DLO | [HSDP](../../docs/user_guide/diffusion/parallelism/hsdp.md) |
|
||||
| CFG parallel | CFG2xSP2 on four workers; CFG2xSP4 requires eight | [CFG parallel](../../docs/user_guide/diffusion/parallelism/cfg_parallel.md) |
|
||||
| VAE patch parallel | TurboVAE tile decode across the complete worker group | [VAE parallelism](../../docs/user_guide/diffusion/parallelism/vae_parallelism.md) |
|
||||
| Cache-DiT | Repeated Preview transformer layers; approximate | [Cache-DiT](../../docs/user_guide/diffusion/cache_acceleration/cache_dit.md) |
|
||||
| One-device offload | Ordinary layerwise DiT plus native auxiliary CPU staging | [CPU offload](../../docs/user_guide/diffusion/cpu_offload.md) |
|
||||
| Quantization | Not supported | [Feature matrix](../../docs/user_guide/diffusion_features.md) |
|
||||
| Ring / pipeline parallel | Not supported | [Feature matrix](../../docs/user_guide/diffusion_features.md) |
|
||||
|
||||
DP request waves must contain exactly `data_parallel_size` requests with the
|
||||
same explicit step count. Data-parallel replicas require TP=1. SP-only DLO
|
||||
cannot AllGather because SP ranks own different MoE-head shards.
|
||||
|
||||
## Request fields
|
||||
|
||||
Common geometry and sampling values use the shared CLI flags. The native
|
||||
Preview pipeline accepts these model-specific `--extra-body` fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `seconds` | Must be `10`. |
|
||||
| `resolution` | `272p` or `540p`; default `540p`. |
|
||||
| `output_width`, `output_height` | Optional final resize; supply both. |
|
||||
| `deterministic` | Must match `MAGI2_DETERMINISTIC` at worker startup. |
|
||||
|
||||
Use `--image` in the shared I2V entrypoint rather than the lower-level
|
||||
`image_path` field.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
ffprobe -v error \
|
||||
-show_entries stream=codec_type,width,height,avg_frame_rate,nb_frames,sample_rate,channels \
|
||||
-of json magi2_540p_t2va.mp4
|
||||
```
|
||||
|
||||
A 540p result contains 125 896x512 frames at 12.5 fps and stereo 44.1 kHz
|
||||
audio. A 272p result contains 125 448x256 frames with the same duration, rate,
|
||||
and audio contract.
|
||||
|
||||
## Local qualification evidence
|
||||
|
||||
### Released-checkpoint E2E
|
||||
|
||||
| Profile | Workload | Steps | E2E | Peak HBM | Peak host PSS | Output |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | --- |
|
||||
| DLO SP4 | 540p T2VA | 100 | 563.855 s | See PR evidence | See PR evidence | `3437d078e...f16fdea` |
|
||||
| DLO SP4 | 540p I2VA | 100 | 556.533 s | See PR evidence | See PR evidence | `b8c2a4ac...ff2de8` |
|
||||
| Layerwise 1 GPU | 272p T2VA | 1 | 8.86 s | 49.29 GiB | Not sampled | Valid video + audio |
|
||||
| Layerwise 1 GPU | 272p T2VA | 4 | 42.55 s | 49.29 GiB | 371.97 GiB | Valid video + audio |
|
||||
|
||||
The one-device four-step host-PSS sampler scans the full process tree and adds
|
||||
observable CPU overhead, so its stage timing is qualification evidence rather
|
||||
than a latency benchmark.
|
||||
|
||||
### Distributed correctness
|
||||
|
||||
- Resident TP4, TP2SP2, and SP4 match a single-rank native oracle within the
|
||||
documented BF16 tolerance; resident SP4 is the reference-aligned default.
|
||||
- Deterministic rank-local DLO SP4 output matched resident SP4 exactly.
|
||||
- HSDP4+SP4, HSDP4+CFG2xSP2, Cache-DiT forced hits, and TurboVAE PP4 have
|
||||
focused four-rank coverage. TurboVAE PP4 matched resident decode exactly.
|
||||
- Full 100-step 540p quality generation is qualified only for four-device DLO
|
||||
SP4. Eight-worker layouts remain configuration-only on this host.
|
||||
|
||||
These are bounded qualification runs, not committed benchmarks. Generated
|
||||
artifacts stay outside the repository, and this integration adds no
|
||||
model-specific example or benchmark script.
|
||||
Reference in new issue
Block a user