anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Model] Add native MAGI-2 Preview diffusion support (#5918)

Browse Source 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu authored and GitHub committed 2026-09-05 15:32:30 +08:00
1 parent 13c617a140
commit d4096e01fe
78 files changed
+9896 -459
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

recipes/README.md
+9 -8
View File
@@ -10,7 +10,8 @@ keep naming and layout consistent, organize recipes by model vendor in a way
that is aligned with
[`vllm-project/recipes`](https://github.com/vllm-project/recipes), but treat
that external repository as a reference for structure rather than the place to
add files for this repo. Use one Markdown file per model family by default.
add files for this repo. Use one Markdown file per model and hardware profile,
named `<model>-<hardware>.md`.
> **Note:** The vLLM-Omni TTS recipes (12 models across 8 providers) are also
> published on the rendered recipes site at
@@ -29,16 +30,16 @@ Example layout:
```text
recipes/
Qwen/
Qwen3-Omni.md
Qwen3-TTS.md
Qwen3-Omni-H100.md
Qwen3-TTS-A100.md
Tencent/
Covo-Audio-Chat.md
Covo-Audio-Chat-L40S.md
```
## Available Recipes
| Recipe | Task | Hardware |
|--------|------|----------|
| -------- | ------ | ---------- |
| [`Baidu/ERNIE-Image.md`](./Baidu/ERNIE-Image.md) | Text-to-image online serving (ERNIE-Image 8B) | 1x or 2x RTX 4090 24GB |
| [`Bagel/BAGEL-7B-MoT.md`](./Bagel/BAGEL-7B-MoT.md) | Text-to-image with shared online/offline examples | 1x A100 80GB / 2x CUDA GPUs |
| [`Boogu/Boogu-Image.md`](./Boogu/Boogu-Image.md) | Text-to-image online serving (Boogu-Image-0.1-Base) | 1x A100/H100 40GB+ |
@@ -62,6 +63,7 @@ recipes/
| [`meituan-longcat/LongCat-Video-Avatar-1.5.md`](./meituan-longcat/LongCat-Video-Avatar-1.5.md) | Audio-driven avatar video generation (AT2V / AI2V, single- and multi-speaker, AVC continuation) | 1x H100 80GB |
| [`mistralai/Voxtral-TTS.md`](./mistralai/Voxtral-TTS.md) | Online serving for TTS | 1x RTX 4090 24GB |
| [`Robbyant/LingBot-Video.md`](./Robbyant/LingBot-Video.md) | Native dense and MoE text-to-video serving | Dense: 1x L20X; MoE: 1x L20X (~67.7 GiB peak) |
| [`SandAI/MAGI-2-preview-L20X.md`](./SandAI/MAGI-2-preview-L20X.md) | Native Preview-stage text/image-to-video-and-audio generation | 1x/4x NVIDIA L20X 140 GiB qualified |
| [`Robbyant/LingBot-World-2.0.md`](./Robbyant/LingBot-World-2.0.md) | Offline and experimental realtime interactive world generation | 1x H200/B200 or 2x B200 for TP=2 |
| [`cosmos3/Cosmos3-Nano.md`](./cosmos3/Cosmos3-Nano.md) | Text-to-image, text-to-video, image-to-video, video-to-video generation, text to video with sound, action policy | 1x H200 141GB / B300 |
| [`cosmos3/Cosmos3-Edge.md`](./cosmos3/Cosmos3-Edge.md) | T2I / T2V / I2V + action (Physical-AI) generation on the Nemotron-based Edge transformer | 1x GPU (~8 GiB) |
@@ -92,8 +94,7 @@ recipes/
| [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB |
| [`JD/JoyAI-VL-Interaction.md`](./JD/JoyAI-VL-Interaction.md) | Real-time streaming video-language interaction (proactive speak/silence/delegate) | 1x GPU 24GB+ |
Within a single recipe file, include different hardware support sections such
as `GPU`, `ROCm`, and `NPU`, and add concrete tested configurations like
`1x A100 80GB` or `2x L40S` inside those sections when applicable.
Keep one accelerator model or platform per recipe file. Add a separately
suffixed recipe and table row for another GPU, ROCm accelerator, or NPU.
See [TEMPLATE.md](./TEMPLATE.md) for the recommended format.
recipes/SandAI/MAGI-2-preview-L20X.md
+288
View File
@@ -0,0 +1,288 @@
# MAGI-2 Preview — NVIDIA L20X
> Native text/image-to-video-and-audio generation with the released Preview stage
## Summary
- Vendor: SandAI
- Model: `sand-ai/MAGI-2-preview`
- Runtime: Native vLLM-Omni pipeline; no SandAI runtime import
- Modes: Offline shared examples and OpenAI-compatible video serving
- Hardware: NVIDIA L20X 140 GiB
- Recommended deployment: Four-GPU resident SP4 (`TP=1`, `SP=4`)
- Maintainer: Community
## Supported model contract
### Tasks and inputs
| Task | Text prompt | Media condition | Shared entrypoint |
| --- | --- | --- | --- |
| T2VA | Required | None | [`text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py) |
| I2VA | Required | Exactly one still image | [`image_to_video.py`](../../examples/offline_inference/image_to_video/image_to_video.py) |
The I2VA path preserves the source aspect ratio and applies MAGI-2's native
resize/pad transform. Video or audio conditions, multiple images, and fused
per-rank request batches are not supported.
### Output
| Category | Supported Preview specification |
| --- | --- |
| Duration | Exactly 10 seconds |
| Video | 125 frames at 12.5 fps |
| Native tiers | `272p` (448x256) and `540p` (896x512) |
| Optional final resize | `output_width` and `output_height` must be supplied together |
| Audio | 44.1 kHz stereo, generated with the video |
| Refiner | The separate 1080p refiner is not included in this integration |
### Profiles provided by this recipe
| Profile | Devices | Purpose | Qualification |
| --- | ---: | --- | --- |
| Resident SP4 | 4 | Recommended fidelity/default deployment | Released smoke and reference parity |
| Rank-local DLO SP4 | 4 | Full-quality lower-HBM transformer streaming | Released checkpoint, 100-step T2VA and I2VA |
| DLO DP4 / DP2SP2 | 4 | Concurrent request throughput | Bounded one- and four-step coverage |
| Ordinary layerwise | 1 | Memory-constrained single request | Released checkpoint, one and four steps |
| Online resident SP4 | 4 | OpenAI-compatible video API | Serving path covered |
## References
- Official model card: <https://huggingface.co/sand-ai/MAGI-2-preview>
- Architecture blog: <https://sand.ai/blog/magi-2-preview>
- Supported models: [`docs/models/supported_models.md`](../../docs/models/supported_models.md)
- Diffusion feature matrix: [`docs/user_guide/diffusion_features.md`](../../docs/user_guide/diffusion_features.md)
## Prepare the checkpoint
The native Preview path uses approximately 274 GiB on disk. The separate
refiner is not required.
```bash
export MAGI2_CKPT_ROOT=/path/to/MAGI-2-preview
hf download sand-ai/MAGI-2-preview \
preview/ text_encoder/ vae/ turbo_vae/ stable-audio-open-1.0/ \
--revision 2dea51b64db47ee5b4402d36fd90829a0c58913b \
--local-dir "$MAGI2_CKPT_ROOT"
```
Install vLLM-Omni and ensure `ffmpeg` is on `PATH`. No second model runtime is
required.
## Hardware support
### Locally qualified NVIDIA configuration
| Item | Value |
| --- | --- |
| Accelerator | NVIDIA L20X |
| Per-device memory | 143,771 MiB (about 140.4 GiB) |
| Device count | 4; the one-device profile uses one of the same GPUs |
| Device interconnect | All-to-all NV18 links (`nvidia-smi topo -m`) |
| Host memory | 2.0 TiB |
The official SandAI runtime requires eight NVIDIA Hopper GPUs. That official
eight-GPU system was unavailable for this PR; compatible eight-worker
topologies pass configuration validation but are not locally runtime-qualified.
No H100, H200, B200, or B300 runtime claim is made here.
### Software environment
| Item | Value |
| --- | --- |
| OS | Ubuntu 22.04.5 LTS, Linux 5.10.134 |
| Python | 3.12.13 |
| NVIDIA driver / CUDA runtime | 570.133.20 / 13.0 |
| PyTorch | 2.11.0+cu130 |
| vLLM | 0.26.0 |
| vLLM-Omni | This PR branch; evidence commits `a5af2a8c` and `42bbfb57` |
| Precision | BF16 |
Set `MAGI2_DETERMINISTIC=1` before worker startup when deterministic kernels
are required. The setting is fixed for the worker lifetime.
Each worker process supports one MAGI-2 pipeline instance. Pipeline startup
sets process-wide deterministic state, so multi-pipeline tests and deployments
must use separate worker processes rather than constructing multiple instances
inside one process.
MAGI-2 uses vLLM's bundled FlashAttention 3 kernels by default on Hopper
(compute capability 9.x), FlashAttention 4 on supported Blackwell devices
with FA3 as the fallback, and FlashAttention 2 on earlier architectures. Set
`MAGI2_FLASH_ATTN_VERSION=2`, `3`, or `4` before worker startup to force an
available kernel for diagnostics or output comparisons.
## Offline generation
### Recommended four-GPU T2VA
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
python examples/offline_inference/text_to_video/text_to_video.py \
--model "$MAGI2_CKPT_ROOT" \
--model-class-name Magi2Pipeline \
--prompt "A red fox walks through fresh snow while wind moves the pine branches." \
--height 512 --width 896 --num-frames 125 \
--num-inference-steps 100 --fps 12.5 \
--tensor-parallel-size 1 --ulysses-degree 4 \
--extra-body '{"seconds":10,"resolution":"540p"}' \
--output magi2_540p_t2va.mp4
```
For a load-and-kernel smoke, use one inference step. A one-step output is not a
quality evaluation.
### Recommended four-GPU I2VA
Use the same geometry and parallelism with the shared I2V entrypoint:
```bash
python examples/offline_inference/image_to_video/image_to_video.py \
--model "$MAGI2_CKPT_ROOT" \
--model-class-name Magi2Pipeline \
--image /path/to/first_frame.png \
--prompt "The fox looks up, then walks forward as snow falls around it." \
--height 512 --width 896 --num-frames 125 \
--num-inference-steps 100 --fps 12.5 \
--tensor-parallel-size 1 --ulysses-degree 4 \
--extra-body '{"seconds":10,"resolution":"540p"}' \
--output magi2_540p_i2va.mp4
```
### One-device layerwise offload
MAGI-2 already stages Qwen, the image VAE, TurboVAE, and Oobleck from pinned
CPU memory. Ordinary layerwise offload streams the Preview DiT blocks so the
complete transformer does not need to fit in HBM at once.
```bash
export CUDA_VISIBLE_DEVICES=0
python examples/offline_inference/text_to_video/text_to_video.py \
--model "$MAGI2_CKPT_ROOT" \
--model-class-name Magi2Pipeline \
--prompt "A red fox walks through fresh snow while wind moves the pine branches." \
--height 256 --width 448 --num-frames 125 \
--num-inference-steps 4 --fps 12.5 \
--enable-cpu-offload --enable-layerwise-offload \
--extra-body '{"seconds":10,"resolution":"272p"}' \
--output magi2_272p_cpu_layerwise.mp4
```
Standalone `--enable-cpu-offload` is rejected because staging whole modules
does not make the complete Preview DiT fit on one qualified GPU. When both
flags are supplied, the shared offloader selects layerwise mode and MAGI's
native auxiliary staging remains active.
## Online serving
The OpenAI-compatible serving defaults are the full `540p` preset and `100`
denoising steps. The model always generates a 10-second, 125-frame clip at
12.5 fps; pass `resolution=272p` and `num_inference_steps=1` for a low-cost
load smoke. Requests with another duration are rejected before generation.
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
vllm serve "$MAGI2_CKPT_ROOT" --omni \
--model-class-name Magi2Pipeline \
--num-gpus 4 \
--tensor-parallel-size 1 \
--ulysses-degree 4 \
--port 8091
```
```bash
curl -X POST http://localhost:8091/v1/videos/sync \
-F 'prompt=A red fox walks through fresh snow.' \
-F 'seconds=10' \
-F 'size=896x512' \
-F 'num_frames=125' \
-F 'fps=12.5' \
-F 'num_inference_steps=100' \
-F 'seed=42' \
-F 'extra_params={"resolution":"540p"}' \
-o magi2_online.mp4
```
For I2VA, add `-F 'input_reference=@first_frame.png;type=image/png'`.
## Supported features
Use the shared guides for launch syntax and feature semantics. The flags below
are the MAGI-2-specific qualified combinations.
| Feature | MAGI-2 status / topology | Guide |
| --- | --- | --- |
| Sequence parallel | Resident SP4 default; compatible SP8 configuration | [Sequence parallel](../../docs/user_guide/diffusion/parallelism/sequence_parallel.md) |
| Tensor parallel | TP4 or TP2SP2 on four workers | [Tensor parallel](../../docs/user_guide/diffusion/parallelism/tensor_parallel.md) |
| DLO | DP4/DP2SP2 AllGather; SP4 rank-local requires `--dlo-no-use-allgather` | [Distributed layerwise offload](../../docs/design/feature/distributed_layerwise_offload.md) |
| HSDP | HSDP4+SP4; alternative to TP and DLO | [HSDP](../../docs/user_guide/diffusion/parallelism/hsdp.md) |
| CFG parallel | CFG2xSP2 on four workers; CFG2xSP4 requires eight | [CFG parallel](../../docs/user_guide/diffusion/parallelism/cfg_parallel.md) |
| VAE patch parallel | TurboVAE tile decode across the complete worker group | [VAE parallelism](../../docs/user_guide/diffusion/parallelism/vae_parallelism.md) |
| Cache-DiT | Repeated Preview transformer layers; approximate | [Cache-DiT](../../docs/user_guide/diffusion/cache_acceleration/cache_dit.md) |
| One-device offload | Ordinary layerwise DiT plus native auxiliary CPU staging | [CPU offload](../../docs/user_guide/diffusion/cpu_offload.md) |
| Quantization | Not supported | [Feature matrix](../../docs/user_guide/diffusion_features.md) |
| Ring / pipeline parallel | Not supported | [Feature matrix](../../docs/user_guide/diffusion_features.md) |
DP request waves must contain exactly `data_parallel_size` requests with the
same explicit step count. Data-parallel replicas require TP=1. SP-only DLO
cannot AllGather because SP ranks own different MoE-head shards.
## Request fields
Common geometry and sampling values use the shared CLI flags. The native
Preview pipeline accepts these model-specific `--extra-body` fields:
| Field | Meaning |
| --- | --- |
| `seconds` | Must be `10`. |
| `resolution` | `272p` or `540p`; default `540p`. |
| `output_width`, `output_height` | Optional final resize; supply both. |
| `deterministic` | Must match `MAGI2_DETERMINISTIC` at worker startup. |
Use `--image` in the shared I2V entrypoint rather than the lower-level
`image_path` field.
## Verification
```bash
ffprobe -v error \
-show_entries stream=codec_type,width,height,avg_frame_rate,nb_frames,sample_rate,channels \
-of json magi2_540p_t2va.mp4
```
A 540p result contains 125 896x512 frames at 12.5 fps and stereo 44.1 kHz
audio. A 272p result contains 125 448x256 frames with the same duration, rate,
and audio contract.
## Local qualification evidence
### Released-checkpoint E2E
| Profile | Workload | Steps | E2E | Peak HBM | Peak host PSS | Output |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| DLO SP4 | 540p T2VA | 100 | 563.855 s | See PR evidence | See PR evidence | `3437d078e...f16fdea` |
| DLO SP4 | 540p I2VA | 100 | 556.533 s | See PR evidence | See PR evidence | `b8c2a4ac...ff2de8` |
| Layerwise 1 GPU | 272p T2VA | 1 | 8.86 s | 49.29 GiB | Not sampled | Valid video + audio |
| Layerwise 1 GPU | 272p T2VA | 4 | 42.55 s | 49.29 GiB | 371.97 GiB | Valid video + audio |
The one-device four-step host-PSS sampler scans the full process tree and adds
observable CPU overhead, so its stage timing is qualification evidence rather
than a latency benchmark.
### Distributed correctness
- Resident TP4, TP2SP2, and SP4 match a single-rank native oracle within the
documented BF16 tolerance; resident SP4 is the reference-aligned default.
- Deterministic rank-local DLO SP4 output matched resident SP4 exactly.
- HSDP4+SP4, HSDP4+CFG2xSP2, Cache-DiT forced hits, and TurboVAE PP4 have
focused four-rank coverage. TurboVAE PP4 matched resident decode exactly.
- Full 100-step 540p quality generation is qualified only for four-device DLO
SP4. Eight-worker layouts remain configuration-only on this host.
These are bounded qualification runs, not committed benchmarks. Generated
artifacts stay outside the repository, and this integration adds no
model-specific example or benchmark script.
recipes/TEMPLATE.md
+40 -36
View File
@@ -1,6 +1,10 @@
# Recipe Title
# Recipe Title — Hardware
> Example: Qwen3-Omni for speech chat
> Example: Qwen3-Omni for speech chat — H100
Name the file `recipes/<vendor>/<model>-<hardware>.md`. Keep exactly one
hardware profile in each recipe; add a separate suffixed recipe when another
accelerator is qualified.
## Summary
@@ -8,28 +12,44 @@
- Model:
- Task:
- Mode:
- Hardware:
- Maintainer:
## When to use this recipe
Briefly describe the concrete scenario this recipe covers.
## Supported model contract
Put user-visible constraints before commands. Use compact tables for:
- supported tasks and shared entrypoints;
- input modalities, counts, duration/geometry limits, and required combinations;
- output duration, geometry, frame/audio rates, and variants;
- the deployment profiles provided on this recipe's named hardware.
Source model specifications from the canonical model card or repository. Do
not copy values from a sibling model or infer support from configuration alone.
## References
- Upstream or canonical docs:
- Related example under `examples/`:
- Related issue or discussion:
## Hardware Support
## Hardware
Add one section per platform, such as `GPU`, `ROCm`, or `NPU`. Under each
platform section, document one or more tested hardware configurations.
- Accelerator model and per-device memory:
- Number of devices:
- Device interconnect (NVLink, PCIe, or other):
- Host memory, when CPU offload or staging is relevant:
- Qualification scope:
## GPU
Do not add a second accelerator to this file. Create another
`<model>-<hardware>.md` recipe instead. Mention unqualified upstream hardware
only as a limitation, not as a provided profile.
### 1x A100 80GB
#### Environment
## Software environment
- OS:
- Python:
@@ -37,7 +57,10 @@ platform section, document one or more tested hardware configurations.
- vLLM version:
- vLLM-Omni version or commit:
#### Command
Keep worker counts, precision, and TP/SP/DP/PP sizes with the command or
deployment profile; they are not software-environment properties.
## Command
Serve commands use `vllm serve <model> --omni`. The `--omni` flag is what
selects the Omni pipeline, so it is required and cannot be dropped.
@@ -46,40 +69,21 @@ selects the Omni pipeline, so it is required and cannot be dropped.
# Add the exact command(s) here
```
#### Verification
## Verification
```bash
# Add a quick validation command or expected output here
```
#### Notes
## Notes
- Memory usage:
- Key flags:
- Known limitations:
### 2x L40S
## Supported features
Repeat the same structure for other hardware setups as needed.
## ROCm
### Example hardware configuration
Repeat the same nested structure for ROCm setups as needed:
- `#### Environment`
- `#### Command`
- `#### Verification`
- `#### Notes`
## NPU
### Example hardware configuration
Repeat the same nested structure for NPU setups as needed:
- `#### Environment`
- `#### Command`
- `#### Verification`
- `#### Notes`
Use a compact table containing the model-specific status/topology and a link to
the corresponding shared feature guide. Do not repeat the general feature
documentation in the recipe. Record unsupported combinations in the same table
instead of a disconnected restrictions section.