汪志鹏andDidan Deng
0059ec878c
Signed-off-by: princepride <wangzhipeng628@gmail.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com> Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
73 lines
2.8 KiB
Markdown
73 lines
2.8 KiB
Markdown
# MagiHuman Generation
|
|
|
|
MagiHuman is an advanced, omni-modality model that generates both high-quality video and lip-synced audio from a text prompt.
|
|
|
|
Because MagiHuman is a very large model featuring a powerful DiT MoE backbone and a ~9B parameter T5Gemma text encoder, it natively supports **Tensor Parallelism (TP)** in vLLM-Omni to run efficiently across multi-GPU setups, reducing device memory bottlenecks.
|
|
|
|
## Setup
|
|
|
|
### Install MagiCompiler (recommended)
|
|
|
|
MagiHuman relies on [MagiCompiler](https://github.com/SandAI-org/MagiCompiler) for custom-op registration used by the DiT attention kernels. While the pipeline can fall back to stub implementations, installing MagiCompiler is **strongly recommended** for correct behaviour.
|
|
|
|
```bash
|
|
# Clone the repo
|
|
git clone https://github.com/SandAI-org/MagiCompiler.git
|
|
cd MagiCompiler
|
|
|
|
# System dependencies (optional, for FX graph visualization; Debian/Ubuntu)
|
|
sudo apt update && sudo apt install -y graphviz
|
|
|
|
# Python dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Install MagiCompiler
|
|
pip install . # end users (recommended)
|
|
# pip install -e . # developers (editable install)
|
|
```
|
|
|
|
### Hardware requirements
|
|
|
|
Ensure your hardware has enough VRAM. For a standard node with 80GB GPUs, running with `--tensor-parallel-size 4` is recommended to shard both the MoE weights and the T5Gemma text encoder across 4 GPUs, reducing the per-GPU peak VRAM overhead significantly (by roughly ~13.5GB per GPU compared to single-device inference).
|
|
|
|
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for further details on allocating memory.
|
|
|
|
## Run Examples
|
|
|
|
Get into the example folder:
|
|
```bash
|
|
cd examples/offline_inference/magi_human
|
|
```
|
|
|
|
### End-to-End Generation (Text to Video+Audio)
|
|
|
|
Generate a video with synchronized speech natively generated by the model.
|
|
|
|
```bash
|
|
python end2end.py \
|
|
--model /proj-tango-pvc/users/zhipeng.wang/workspace/models/daVinci-MagiHuman \
|
|
--prompt "A young woman with long, wavy golden blonde hair..." \
|
|
--tensor-parallel-size 4 \
|
|
--output output_magihuman.mp4
|
|
```
|
|
|
|
## Common Parameters
|
|
|
|
| Parameter | Default | Description |
|
|
|-----------|---------|-------------|
|
|
| `--model` | *(Required)* | Local model path or HuggingFace ID |
|
|
| `--prompt` | *(built-in demo prompt)* | Highly detailed text prompt dictating visual look and dialogue text |
|
|
| `--tensor-parallel-size` | `4` | Tensor parallelism size (Number of GPUs) |
|
|
| `--height` | `256` | Initial resolution height |
|
|
| `--width` | `448` | Initial resolution width |
|
|
| `--num-inference-steps` | `8` | Denoising steps |
|
|
| `--seed` | `52` | Random seed |
|
|
| `--output` | `output_magihuman.mp4` | Output video with audio path |
|
|
|
|
## Example materials
|
|
|
|
??? abstract "end2end.py"
|
|
``````py
|
|
--8<-- "examples/offline_inference/magi_human/end2end.py"
|
|
``````
|