anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Doc][NPU] Refresh MiniMax-H3 Ascend recipes for 0.28-era configurations (#7720)

Browse Source 
Signed-off-by: brandneway <gyuan4892@gmail.com>
This commit is contained in:
ooooooye authored and GitHub committed 2026-09-17 11:05:15 +00:00
1 parent bf74544a35
commit 873e9bff7c
2 files changed
+477 -316
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

recipes/MiniMaxAI/MiniMax-H3-NPU-950PR.md
+183 -154
View File
@@ -1,195 +1,224 @@
# MiniMax-H3 on a single NPU 950PR
# MiniMax H3 on Ascend 950PR / 950DT
This recipe runs MiniMax-H3 with online INT8 quantization on one NPU 950PR
NPU (128 GB HiBL 1.0 HBM). It covers the single-card T2VA configuration at
1024x576. For the eight-card Atlas 800I A3 BF16 route at 768P, see
[MiniMax-H3-NPU.md](MiniMax-H3-NPU.md).
> Joint video and audio generation — Ascend NPU deployment guide for the
> four-card Ascend 950PR / 950DT route
## Capacity requirements
## Summary
| Resource | Requirement |
| --- | ---: |
| NPU | 1x NPU 950PR |
| NPU HBM | 128 GiB (131,072 MiB reported by `npu-smi`) |
| Observed HBM high-water mark | 118,442 MiB (90.4% of capacity) |
| Checkpoint storage | 135 GiB per partition, local disk strongly preferred |
| Container shared memory | 8 GiB, or a writable local filesystem for spill |
The observed high-water mark leaves about 12.3 GiB of headroom, so this card is
sized for exactly one resident task partition at 1024x576. Do not expect a
larger output shape or concurrency greater than one to fit.
Because a single 950PR holds the whole partition in HBM, this route does not
need layerwise offload, and the 200 GiB system-RAM floor carried by the 72 GiB
GPU recipes does not apply. The validated run completed inside a container
limited to 32 GiB of RAM. Passing `--enable-layerwise-offload` on this card is
actively harmful: it stages weights in non-reclaimable host memory and the
container's OOM killer terminates the server with exit code -9.
- Vendor: MiniMaxAI
- Model: [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)
- Tasks: T2VA, FL2VA, and Ref2VA
- Mode: OpenAI-compatible `/v1/videos` HTTP serving
- Hardware: Ascend 950PR / Ascend 950DT, 4x NPU (128 GB HBM per device)
- Maintainer: Community
This recipe covers the four-card 950PR / 950DT configuration at 1344x768.
For the eight-card Atlas 800I A2 / A3 route, see
[MiniMax-H3-NPU.md](MiniMax-H3-NPU.md); checkpoint layout, container
preparation, MindIE-SD installation, and the ffmpeg/decord dependencies are
described there and apply unchanged.
## Environment
- Host architecture: x86_64
- driver: 25.7.rc1.6 (hal 7.35.23)
- firmware: 9.0.0.105.229
- CANN toolkit: 9.1.0 (`/usr/local//cann-9.1.0`)
- npu-smi: 25.7.rc1.6
- Python: 3.12.13
- PyTorch: 2.10.0+cpu
- torch_npu: 2.10.0.post2
- vLLM: 0.26.0
- vLLM-Omni: 0.26.1.dev103+g584d78c67.npu (commit `584d78c6`)
Install vLLM-Omni from a checkout with MiniMax-H3 support:
Use the A5 variant of the official vLLM-Omni NPU images (see
[NPU installation](../../docs/getting_started/installation/npu.md)):
```bash
uv venv
source .venv/bin/activate
uv pip install -e .
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-a5
```
Check the [tag list](https://quay.io/repository/ascend/vllm-omni?tab=tags)
for newer releases. Inside the container, install MindIE-SD and the optional
Ref2VA media dependencies as described in
[MiniMax-H3-NPU.md § Environment](MiniMax-H3-NPU.md#environment).
## Start a server
Task selection follows the eight-card recipe: pass `--task-type fl2va` or
`--task-type ref2va` to pick the task partition. The configurations below use
4-card USP, 4-card text-encoder TP, and VAE `tile` parallelism.
The two recommended configurations differ in how they fit memory: the
lossless configuration enables distributed layerwise offload (DLO) to avoid
activation OOM at longer durations, while the lossy configuration fits the
maximum supported MiniMax-H3 workload shape without OOM, so DLO is omitted
for performance.
### Recommended lossless configuration (Ascend 950PR / 950DT)
```bash
export MODEL=/path/to/MiniMax-H3/FL2VA
export PORT=8000
export PORT=9098
export MODEL=/path/to/MiniMax-H3
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400
export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=4000
export PYTHONDONTWRITEBYTECODE=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_NPU_SOCKET_PORT_RANGE="auto"
vllm serve "${MODEL}" \
--omni \
--host 0.0.0.0 \
--port "${PORT}" \
--trust-remote-code \
--num-gpus 1 \
--tensor-parallel-size 1 \
--usp 1 \
--task-type fl2va \
--num-gpus 4 \
--usp 4 \
--ring 1 \
--text-encoder-tp-size 1 \
--vae-patch-parallel-size 1 \
--text-encoder-tp-size 4 \
--vae-parallel-mode tile \
--vae-use-tiling \
--init-timeout 14400 \
--stage-init-timeout 14400 \
--quantization int8
--vae-patch-parallel-size 4 \
--enable-diffusion-pipeline-profiler \
--enable-distributed-layerwise-offload \
--diffusion-attention-backend FLASH_ATTN
```
Do not pass `--diffusion-attention-backend CUDNN_ATTN`. cuDNN attention has no
NPU implementation and fails at the first denoise step. Leave the backend
unset: it resolves to `TORCH_SDPA`, or to `FLASH_ATTN` when MindIE-SD is
installed (see [MiniMax-H3-NPU.md § Environment](MiniMax-H3-NPU.md#environment)).
H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1. The first
request includes regional compilation; warm the server once before measuring
steady-state latency.
H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.
### Recommended lossy configuration (Ascend 950PR / 950DT)
## Validated evidence
Measured on one NPU 950PR with the server command above plus
`--enable-diffusion-pipeline-profiler`, generating a 5 s 1024x576 T2VA clip from
the request in [§ T2VA request example](#t2va-request-example). MiniMax-H3
requested 60 denoise steps and executed 59 denoise updates, so per-step latency
is `denoise / 59`. `ffprobe` confirms every returned MP4 is 1024x576, 24 fps,
124 video frames, 5.175 s, with a 32 kHz audio track of 165,600 samples.
Five requests were issued back to back with no idle time between them. The card
is measurably faster on the first request than in steady state, so both regimes
are reported.
| Stage | First request, cold card | Thermal steady state |
| --- | ---: | ---: |
| End-to-end request | 462.13 s | 506.38 s |
| Text encode | 0.55 s | 0.05 s |
| Denoise (59 updates) | 429.29 s | 471.74 s |
| Per denoise update | 7,276 ms | 7,996 ms |
| VAE decode | 8.43 s | 8.14 s |
| Worker-to-server handoff | 24 s | 27 s |
| MP4 muxing | 0.57 s | 0.50 s |
| Server-reported `denoise_step_latency_ms` | 7,702 ms | 8,440 ms |
| Peak HBM (`npu-smi`) | 118,442 MiB | 118,442 MiB |
| Average NPU power | 566.9 W | 556.9 W |
| Peak NPU temperature | 104 C | 104 C |
Sample counts differ per row. The cold column is a single request. In the steady
column, text encode, denoise and VAE decode are means over the four subsequent
requests — denoise spanned 470.37 to 473.21 s, a 0.60% spread — while
end-to-end, handoff and muxing are means over the two of those four that
survived the handoff timeout described above.
**Report the steady-state column.** The cold card completes denoise 9.9% faster
while drawing only 1.8% more average power, and both regimes reach the same
104 C ceiling, so the first request is buying a short window of higher clocks
before the die saturates. Any benchmark that issues a single request against an
idle card will overstate throughput by about 10%.
**Do not read `denoise_step_latency_ms` as a per-step time.** The server divides
the entire stage wall time by the 60 *requested* steps, so it absorbs text
encoding, VAE decode and the 837 MiB handoff. All three successful requests
satisfy `denoise_step_latency_ms == e2e_stage_wall_time_ms / 60` exactly. The
per-update figure in the table is `diffuse / 59` from the pipeline profiler.
Denoise accounts for 93% of the steady-state request and the handoff accounts
for 5.3%, so keeping the transfer in shared memory should bring end-to-end down
to roughly 480 s.
Peak HBM is sampled externally with `npu-smi` at a 3 s interval. It is a
reserved high-water mark rather than live allocation: it reads 111,272 MiB after
weight loading and before the first request, and does not fall back after
requests complete. Re-measure for longer outputs, a different output shape, or
concurrency greater than one.
### Startup
Weight loading took **31 min 50 s** on the validated host (first log line
08:04:59, `Application startup complete` 08:36:49), with the checkpoint on a
shared GlusterFS network mount. This is why the server command sets
`--init-timeout 14400` and `--stage-init-timeout 14400`; the stock 600 s and
1800 s timeouts both abort mid-load with `TimeoutError` and exit code 143.
Reduce both timeouts when the partition is staged on local disk.
## T2VA request example
Adds EQBSA sparse attention (block-sparse with Q/K INT8 + V FP8 mixed
precision) and MXFP8 online quantization, and drops DLO:
```bash
export API_URL="http://127.0.0.1:${PORT}/v1/videos/sync"
export PORT=9098
export MODEL=/path/to/MiniMax-H3
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=4000
export PYTHONDONTWRITEBYTECODE=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_NPU_SOCKET_PORT_RANGE="auto"
curl -sS --max-time 1800 -X POST "${API_URL}" \
-F 'prompt=At night, three cats march into a bedroom playing tiny brass instruments, then abruptly file out, with synchronized room ambience.' \
-F 'width=1024' \
-F 'height=576' \
-F 'aspect_ratio=16:9' \
-F 'fps=24' \
-F 'num_inference_steps=60' \
-F 'flow_shift=12' \
-F 'seed=1101' \
-F 'extra_params={"task":"t2va","duration":5,"audio_flow_shift":3.0}' \
-o t2va.mp4
vllm serve "${MODEL}" \
--omni \
--host 0.0.0.0 \
--port "${PORT}" \
--trust-remote-code \
--task-type fl2va \
--num-gpus 4 \
--usp 4 \
--ring 1 \
--text-encoder-tp-size 4 \
--vae-parallel-mode tile \
--vae-use-tiling \
--vae-patch-parallel-size 4 \
--enable-diffusion-pipeline-profiler \
--diffusion-attention-config '{"default":{"backend":"RAINFUSION_ATTN","block_sparse":{"sparsity":0.8,"precision":"mix","start_step":8,"end_step":12}}}' \
--diffusion-quantization-config '{"transformer":{"method":"mxfp8"}}'
```
The recommended lossy configuration uses a **start=8 / end=12** dense-fallback
window; see below for more aggressive settings.
### Optional optimizations
The following sections list only the **increment or replacement** relative to
the recommended lossless configuration; environment variables and the
remaining flags stay unchanged.
#### Tuning the EQBSA dense-fallback window
`start_step` and `end_step` are the numbers of leading/trailing steps that
fall back to dense attention — not step indices. `precision` defaults to
`bf16`; EQBSA requires setting it to `mix` explicitly. For 1344x768, 50-step
T2VA:
| Configuration | `start_step` | `end_step` | Notes |
| --- | ---: | ---: | --- |
| Quality first (recommended) | 8 | 13 | General and complex-motion scenes |
| Balanced | 0 | 13 | Balanced |
| Speed first | 0 | 0 | Simple, low-motion scenes |
When frames look discontinuous or details unstable, increase `end_step`
first, then `start_step`; do not compensate by lowering `sparsity` alone.
These pairings were validated at 1344x768 / 50 steps only; re-evaluate for
other resolutions or step counts.
#### Cache-DiT
Append to enable DiT block caching with TaylorSeer extrapolation — see the
[Cache-DiT guide](../../docs/user_guide/diffusion/cache_acceleration/cache_dit.md):
```bash
--cache-backend cache_dit \
--enable-cache-dit-summary \
--cache-config '{"Fn_compute_blocks":2,"Bn_compute_blocks":1,"max_warmup_steps":4,"residual_diff_threshold":0.4,"max_continuous_cached_steps":4,"enable_taylorseer":true,"taylorseer_order":2}'
```
#### MXFP4 online quantization
Append to switch the DiT to MXFP4 online quantization:
```bash
--diffusion-quantization-config '{"transformer":{"method":"mxfp4"}}'
```
and remove `--enable-distributed-layerwise-offload`.
## Request examples
Identical to the eight-card recipe; see
[MiniMax-H3-NPU.md § HTTP API examples](MiniMax-H3-NPU.md#http-api-examples)
and the GPU recipe's full parameter table.
## Benchmarks (Ascend 950PR)
Measured with `--enable-diffusion-pipeline-profiler` on vLLM-Omni 0.28.0:
lossless rows use the recommended lossless configuration (with DLO), lossy
rows the recommended lossy configuration (EQBSA + MXFP8, without DLO). Memory
figures are per device.
| Task | Resolution | Frames | Duration (s) | Config | E2E (s) | DiT total (s) | DiT per-step (s) | DiT steps | Text encode (s) | Ref video encode (s) | Ref audio encode (s) | VAE decode (s) | Ref preprocess (s) | Post-process (s) | CPU MP4 (s) | Resident weights (GB) | Peak memory (GB) |
| ---- | ---- | ---- | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| t2va | 1344x768 | 124 | 5 | lossless | 203.95 | 198.10 | 4.04 | 49 | 0.57 | NA | NA | 5.05 | NA | 0.001 | 0.23 | 3.97 | 22.2 |
| t2va | 1344x768 | 362 | 15 | lossless | 1489.17 | 1474.35 | 30.08 | 49 | 0.69 | NA | NA | 13.45 | NA | 0.001 | 0.82 | 3.97 | 32.14 |
| t2va | 1344x768 | 124 | 5 | lossy | 113.86 | 109.81 | 2.24 | 49 | 0.03 | NA | NA | 3.82 | NA | 0.001 | 0.21 | 56.99 | 64.38 |
| t2va | 1344x768 | 362 | 15 | lossy | 762.17 | 749.40 | 15.29 | 49 | 0.03 | NA | NA | 12.19 | NA | 0.001 | 0.61 | 56.99 | 74.26 |
| ref2va | 1344x768 | 124 | 5 | lossless | 745.43 | 730.22 | 14.9 | 49 | 1.16 | 8.15 | 0.13 | 5.12 | 0.41 | 0.001 | 0.28 | 3.97 | 23.82 |
| ref2va | 1344x768 | 362 | 15 | lossless | 5887.24 | 5847.57 | 119.33 | 49 | 2.54 | 22.07 | 0.23 | 13.34 | 0.81 | 0.001 | 0.70 | 3.97 | 32.53 |
| ref2va | 1344x768 | 124 | 5 | lossy | 400.40 | 387.82 | 7.91 | 49 | 0.95 | 7.05 | 0.09 | 3.91 | 0.41 | 0.001 | 0.21 | 56.6 | 66.03 |
| ref2va | 1344x768 | 362 | 15 | lossy | 3037.84 | 3001.31 | 61.24 | 49 | 2.28 | 20.64 | 0.15 | 12.1 | 0.81 | 0.001 | 0.70 | 56.6 | 74.76 |
The resident-weight figures reflect the DLO strategy difference: with DLO
(lossless rows) DiT weights live in host memory and only the active layer
stays on the device, while without DLO (lossy rows) the quantized weights are
resident in HBM. Peak memory stays well inside the 128 GB per-device capacity
in both cases. These numbers describe the validated shapes rather than a
general throughput guarantee.
## Known limitations
- INT8 online quantization is validated for T2VA on this card. Use BF16 for
FL2VA and Ref2VA, or re-measure before relying on it.
- Single-card serving loads one task partition at a time. For Ref2VA, stop the
server and restart it against the `Ref2VA` directory.
- Sustained load drives the die to 104 C and costs about 10% of denoise
throughput relative to a cold card. Warm the server with at least one full
request before measuring, and treat steady state as the reportable number.
- Requests can fail during result handoff on hosts where the IPC spill path is
slow. See [§ Result handoff and shared
memory](#result-handoff-and-shared-memory).
- Loading the checkpoint from a network mount takes over 30 minutes and forces
very large init timeouts. Stage the partition on local disk when possible.
- The configuration measured here is 1024x576 at 60 steps, which differs from
the 1344x768 at 50 steps used by the GPU recipes in this directory. Denoise
cost scales superlinearly with token count, so the numbers are not directly
comparable across recipes.
- The image ships a `triton` package whose backend is not built
(`No module named 'triton._C.libtriton.'`). vLLM logs this as an error
at startup and disables Triton; the diffusion path does not need it and the
run completes normally.
- Task serving is partitioned by `--task-type`: T2VA/FL2VA and Ref2VA load
different DiTs, so switching between them requires a restart with the other
partition.
- H3 currently executes one generation request per diffusion batch.
- The first regional-compile request is a warmup and should not be included
in steady-state performance measurements.
- VAE patch parallelism requires size 1 or the full DiT group size and
supports the H3 native `tile` mode only.
- The lossy configuration (EQBSA + MXFP8) is validated for T2VA and Ref2VA;
use the lossless configuration for FL2VA or re-validate first.
- The lossless configuration trades throughput for activation headroom via
DLO; if your workload never approaches the maximum duration, dropping
`--enable-distributed-layerwise-offload` may improve latency at the risk of
OOM on longer durations.
## Additional resources
- [MiniMax-H3.md](MiniMax-H3.md) — full GPU guide
- [MiniMax-H3-NPU.md](MiniMax-H3-NPU.md) — eight-card Atlas 800I A3 BF16 guide
- [Int8 quantization](../../docs/user_guide/quantization/int8.md)
- [MiniMax-H3-NPU.md](MiniMax-H3-NPU.md) — eight-card Atlas 800I A2 / A3
guide (checkpoint, container, and dependency details)
- [NPU installation](../../docs/getting_started/installation/npu.md)
- [RainFusion attention](../../docs/user_guide/diffusion/attention_backends/rainfusion.md)
— block-sparse knobs and tuning
- [Online quantization](../../docs/user_guide/quantization/online.md)
— INT8 / MXFP8 / MXFP4
- [Cache-DiT guide](../../docs/user_guide/diffusion/cache_acceleration/cache_dit.md)
- [Supported models](../../docs/models/supported_models.md)
- [Video API](../../docs/serving/videos_api.md)
recipes/MiniMaxAI/MiniMax-H3-NPU.md
+294 -162
View File
@@ -1,7 +1,8 @@
# MiniMax H3 on Ascend NPU
> Joint video and audio generation with text, first/last keyframes, and mixed
> image/video/audio references — Ascend NPU deployment guide
> image/video/audio references — Ascend NPU deployment guide for Atlas 800I A2
> and Atlas 800I A3
## Summary
@@ -9,7 +10,7 @@
- Model: [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)
- Tasks: T2VA, FL2VA, and Ref2VA
- Mode: OpenAI-compatible `/v1/videos` HTTP serving
- Hardware: Atlas 800I A3
- Hardware: Atlas 800I A2 / Atlas 800I A3, 8x NPU
- Maintainer: Community
This recipe adapts [MiniMax-H3.md](MiniMax-H3.md) for Ascend NPU
@@ -22,103 +23,288 @@ environments. Differences from the GPU path:
automatically falls back to soundfile / ffmpeg for wav/mp3/m4a/mp4 audio
inputs.
For the four-card Ascend 950PR / 950DT route, see
[MiniMax-H3-NPU-950PR.md](MiniMax-H3-NPU-950PR.md).
## Prerequisites
### Checkpoint
Same as the GPU recipe — Hugging Face access approval is required. Authenticate
once; `vllm serve` downloads the required nested components automatically:
Download the weights once; `vllm serve` accepts the repository ID or a local
path and resolves the nested components automatically:
```bash
hf auth login
export MODEL=MiniMaxAI/MiniMax-H3
export MODEL=/path/to/MiniMax-H3
hf download MiniMaxAI/MiniMax-H3 --local-dir "${MODEL}"
```
The checkpoint directory layout is:
```text
MiniMax-H3/
├── FL2VA/ # shared by T2VA/FL2VA tasks
├── Ref2VA/ # Ref2VA task weights
└── model_index.json
```
### Environment
- Ascend driver & firmware: 25.5.1
- CANN toolkit: 9.0.1
- Python: 3.12
- PyTorch: 2.10.0+cpu
- torch_npu: 2.10.0.post2
- Install vLLM-Omni from a checkout with MiniMax H3 support:
The recommended deployment uses the official vLLM-Omni NPU images, which ship
the matching vLLM / vLLM-Ascend / vLLM-Omni stack (see
[NPU installation](../../docs/getting_started/installation/npu.md)). Pick the
image matching the hardware — `quay.io/ascend/vllm-omni:v0.29.0` for
Atlas 800I A2, `quay.io/ascend/vllm-omni:v0.29.0-a3` for Atlas 800I A3 — and
check the
[tag list](https://quay.io/repository/ascend/vllm-omni?tab=tags) for newer
releases:
```bash
uv venv
source .venv/bin/activate
uv pip install -e .
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-a3
export CONTAINER_NAME=h3
docker run -it -u root --name ${CONTAINER_NAME} \
--privileged=true \
--shm-size=2000g \
--net=host \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /home:/home \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
${IMAGE} \
/bin/bash
```
Install the **mindie-sd** operator library to enable Ascend-optimized fused
operators (`adalayernorm`, etc.) and the RainFusion `rf_v2` block-sparse
attention kernel:
Inside the container:
- **MindIE-SD** provides the Ascend-optimized fused operators
(`adalayernorm`, etc.), the LaserAttention kernel, and the RainFusion
block-sparse attention kernel. If the image does not ship it, build it from
source:
```bash
git clone https://gitcode.com/Ascend/MindIE-SD.git && cd MindIE-SD
# Comment out the tik_ops build step (not needed for this use case)
sed -i 's|^\(\s*\)source ${current_script_dir}/build_tik_ops.sh|\1# source ${current_script_dir}/build_tik_ops.sh|' build/build_ops.sh
git clone https://gitcode.com/Ascend/MindIE-SD.git
cd MindIE-SD
python setup.py bdist_wheel
cd dist
pip install mindiesd-*.whl
pip install dist/mindiesd-*.whl
```
- `ffmpeg` and `ffprobe` must be available on `PATH`. They are used for
reference-video preparation and MP4 output.
- Reference-video decoding uses `decord` when available and falls back to
PyAV otherwise.
- Audio inputs do not require TorchCodec: wav/mp3/m4a/mp4 files are loaded
at native sample rate through the soundfile / ffmpeg fallback.
- To track the latest vLLM-Omni, install it from a checkout:
```bash
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
VLLM_OMNI_TARGET_DEVICE=npu pip install -e . --no-build-isolation
```
- `ffmpeg` and `ffprobe` must be available on `PATH` (reference-video
preparation and MP4 output). Reference-video decoding uses `decord` when
available and falls back to PyAV otherwise; `decord` is required only for
the Ref2VA workflow and can be built with:
```bash
apt update
apt install -y ffmpeg pkg-config libavcodec-dev libavformat-dev \
libavutil-dev libswscale-dev libavfilter-dev libavdevice-dev
git clone --depth 1 --recursive https://github.com/dmlc/decord.git
cd decord
mkdir -p build && cd build
cmake .. -DUSE_CUDA=0 -DDECODE_FFMPEG=1 -DCMAKE_BUILD_TYPE=Release
make
cd ../python
pip3 install . --no-build-isolation
```
## Start a server
Pass the repository ID directly. The pipeline loads the two nested DiTs while
sharing the tokenizer, processor, text encoder, and VAEs from `FL2VA`.
T2VA and FL2VA share one set of weights; the Ref2VA weights differ. Select
the task partition at startup with **`--task-type`** (`fl2va` or `ref2va`).
When unset, only the T2VA/FL2VA weights are loaded. All remaining flags are
task-agnostic.
### Multi-NPU: 768P combined-service configuration
### Recommended lossless configuration (Atlas 800I A2 / A3)
Use Ulysses sequence parallelism degree 8, text-encoder tensor parallelism
degree 8, native tiled VAE patch parallelism degree 8, and distributed
layerwise offload:
Lossless means fused operators plus the high-performance FLASH_ATTN backend
only. The configuration uses 8-card USP, 8-card text-encoder TP, distributed
layerwise offload (DLO), and VAE `tile` parallelism:
```bash
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PORT=9098
export MODEL=MiniMaxAI/MiniMax-H3
export MODEL=/path/to/MiniMax-H3
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800
export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=4000
export PYTHONDONTWRITEBYTECODE=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export MINDIE_SD_FA_TYPE="ascend_laser_attention"
export HCCL_NPU_SOCKET_PORT_RANGE="auto"
vllm serve "${MODEL}" \
--omni \
--host 0.0.0.0 \
--port "${PORT}" \
--trust-remote-code \
--task-type fl2va \
--num-gpus 8 \
--usp 8 \
--ring 1 \
--text-encoder-tp-size 8 \
--enable-distributed-layerwise-offload \
--vae-parallel-mode tile \
--vae-use-tiling \
--vae-patch-parallel-size 8 \
--enable-distributed-layerwise-offload \
--enable-diffusion-pipeline-profiler \
--diffusion-attention-backend FLASH_ATTN
```
Do not add `--enforce-eager`. The first request includes regional
compilation; warm the server once before measuring steady-state latency.
H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.
Notes:
The same endpoint accepts `task=t2va`, `task=fl2va`, and `task=ref2va`; no
partition restart is required. Layerwise offload applies to both DiTs.
- Do not add `--enforce-eager`. The first request includes regional
compilation; warm the server once before measuring steady-state latency.
- H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.
- Keep `--ring 1` when switching to RainFusion attention (below): the
block-sparse kernel ranks key blocks over the whole sequence, so ring
parallelism would split away the keys it needs. Scale with `--usp` instead.
- DLO offloads DiT layers with parameters gathered across the parallel group
instead of replicated per rank, which is what fits the combined 768P
service into 64 GB HBM per device (Atlas 800I A3). Host (CPU) memory usage
is high with this option.
On Atlas 800I A3 (64 GB HBM per device) the combined service does not fit at
768P without offloading or sharding: use distributed layerwise offload (as
above) or HSDP — see
[§ Memory and attention optimizations](#memory-and-attention-optimizations-a3).
### Recommended lossy configuration (Atlas 800I A2 / A3)
### CPU MP4 response encoding (Atlas A2)
Adds RainFusion block-sparse attention and INT8 online quantization on top of
the lossless configuration. Validated for T2VA and Ref2VA; see
[Benchmarks](#benchmarks-atlas-800i-a2--a3):
```bash
vllm serve "${MODEL}" \
--omni \
--host 0.0.0.0 \
--port "${PORT}" \
--trust-remote-code \
--task-type fl2va \
--num-gpus 8 \
--usp 8 \
--ring 1 \
--text-encoder-tp-size 8 \
--vae-parallel-mode tile \
--vae-use-tiling \
--vae-patch-parallel-size 8 \
--enable-distributed-layerwise-offload \
--enable-diffusion-pipeline-profiler \
--diffusion-quantization-config '{"transformer":{"method":"int8"}}' \
--diffusion-attention-config '{"default": {"backend": "RAINFUSION_ATTN",
"block_sparse": {"sparsity": 0.8, "start_step": 12}}}'
```
Online INT8 quantization can be combined with distributed layerwise offload
while AllGather is enabled. Keep AllGather enabled for SP>1; use
`--dlo-no-use-allgather` only when collective communication is unavailable.
### Optional optimizations
The following sections list only the **increment or replacement** relative to
the recommended lossless configuration; environment variables and the
remaining flags stay unchanged.
#### Cache-DiT
Append to enable DiT block caching with TaylorSeer extrapolation — see the
[Cache-DiT guide](../../docs/user_guide/diffusion/cache_acceleration/cache_dit.md):
```bash
--cache-backend cache_dit \
--enable-cache-dit-summary \
--cache-config '{"Fn_compute_blocks":2,"Bn_compute_blocks":1,"max_warmup_steps":4,"residual_diff_threshold":0.4,"max_continuous_cached_steps":4,"enable_taylorseer":true,"taylorseer_order":2}'
```
#### EQBSA sparse attention
EQBSA applies Q/K INT8 with V FP8 mixed quantization on top of block-sparse
attention. Replace `--diffusion-attention-backend FLASH_ATTN` with:
```bash
--diffusion-attention-config '{"default":{"backend":"RAINFUSION_ATTN","block_sparse":{"sparsity":0.8,"start_step":8,"end_step":13,"precision":"mix"}}}'
```
`precision` defaults to `bf16`; enabling EQBSA requires setting it to `mix`
explicitly. `start_step` and `end_step` are the numbers of leading/trailing
steps that fall back to dense attention — not step indices. For 1344x768,
50-step T2VA:
| Configuration | `start_step` | `end_step` | Notes |
| --- | ---: | ---: | --- |
| Quality first (recommended) | 8 | 13 | General and complex-motion scenes |
| Balanced | 0 | 13 | Balanced |
| Speed first | 0 | 0 | Simple, low-motion scenes |
When frames look discontinuous or details unstable, increase `end_step`
first, then `start_step`; do not compensate by lowering `sparsity` alone.
These pairings were validated at 1344x768 / 50 steps only; re-evaluate for
other resolutions or step counts.
## HTTP API examples
The request contract is identical to the GPU recipe; see
[MiniMax-H3.md § HTTP API examples](MiniMax-H3.md#http-api-examples). Use the
validated 768P shapes (e.g. `width=1344 height=768`) on NPU. Quick start
against the synchronous endpoint:
```bash
export API_URL="http://127.0.0.1:${PORT}/v1/videos/sync"
curl -sS -X POST "${API_URL}" \
-F 'prompt=In a snowy blue-purple forest, Ori carefully walks past a sleeping giant; footsteps crunch in the snow while the creature breathes and softly snorts.' \
-F 'width=1344' \
-F 'height=768' \
-F 'aspect_ratio=16:9' \
-F 'fps=24' \
-F 'num_inference_steps=50' \
-F 'flow_shift=12.0' \
-F 'audio_flow_shift=3.0' \
-F 'seed=1101' \
-F 'extra_params={"task":"t2va","duration":5.0}' \
-o t2va.mp4
```
Switch tasks via `extra_params.task` (`t2va` / `fl2va` / `ref2va`); FL2VA and
Ref2VA additionally pass `input_references` (first-frame image, reference
video/audio). The full parameter table lives in the GPU recipe.
## FlashGen 4-step online LoRA (T2VA)
The FlashGen 4-step weights are a native-layout LoRA single file
(`key_format=minimax-h3-native`) loaded at runtime — no merge into the base
checkpoint. Download them from
[FlashGen/Minimax-H3-4step-lora-flashgen](https://modelscope.cn/models/FlashGen/Minimax-H3-4step-lora-flashgen),
then on top of the recommended configuration above:
- append `--lora-backend peft --lora-path "${FLASHGEN_LORA}"`,
- use `--diffusion-attention-backend FLASH_ATTN` (dense; drop the RainFusion
attention config),
- issue T2VA requests with `num_inference_steps=4` and the `lora` request
field (`name` / `path` / `scale`) — see
[MiniMax-H3.md § FlashGen native LoRA](MiniMax-H3.md#flashgen-native-lora).
T2VA only; FL2VA and Ref2VA are not supported, and neither
`--enable-cpu-offload` nor `--enable-layerwise-offload` may be combined with
it.
## CPU MP4 response encoding (Atlas A2)
Non-streaming MP4 responses use one public automatic encoder. It checks the
runtime frame shape, common dtype, and RGB channel-plane contiguity for every
@@ -127,121 +313,65 @@ inputs fall back to the legacy muxer before opening the PyAV container. No CLI
flag, model declaration, or user configuration is required. Streaming fMP4
output remains on its existing incremental path.
Current performance and correctness validation is limited to one Atlas A2 host
with 8x Ascend 910B4-1 NPUs, the `FL2VA`/`t2va` partition, one request at a
time, 1344x768 at 24 fps, and 5, 8.7, and 15 second requests. This optimization
only changes CPU MP4 response encoding; it does not change DiT execution or
stage 0.
## Ascend-optimized components
### Optional optimizations
The optimizations stacked by the recommended configurations above:
Two independent optimizations may be enabled on top of the configuration
above. Both are validated for T2VA only.
1. Ascend-friendly attention: the MindIE-SD `ascend_laser_attention` fused
kernel selected through `MINDIE_SD_FA_TYPE`, carried by the FLASH_ATTN
backend with a mask-free packed varlen path and K/V prefix slicing driven
by the packed `cu_seqlens` metadata.
2. Ascend-friendly fused operators: RMSNorm, AddRMSNorm, GQA, SwiGLU, and
rotary position embedding from MindIE-SD.
3. RainFusion block-sparse attention, optionally with EQBSA mixed INT8/FP8
precision — see
[RainFusion](../../docs/user_guide/diffusion/attention_backends/rainfusion.md).
4. Online quantization: INT8, MXFP8, and MXFP4 — see
[online quantization](../../docs/user_guide/quantization/online.md).
5. Parallelism: USP, text-encoder TP, and native VAE `tile` patch
parallelism, plus distributed layerwise offload.
6. Post-training: few-step distillation via the FlashGen online LoRA.
**RainFusion block-sparse attention** — switch the attention backend, keeping
every other flag unchanged:
## Benchmarks (Atlas 800I A2 / A3)
```bash
--diffusion-attention-backend RAINFUSION_ATTN
```
Measured with `--enable-diffusion-pipeline-profiler` on vLLM-Omni 0.28.0:
lossless rows use the recommended lossless configuration, lossy rows the
recommended lossy configuration. Memory figures are per device.
**INT8 online quantization** — add one flag to the server command above:
### Atlas 800I A2
```bash
--quantization int8
```
| Task | Resolution | Frames | Duration (s) | Config | E2E (s) | DiT total (s) | DiT per-step (s) | DiT steps | Text encode (s) | Ref video encode (s) | Ref audio encode (s) | VAE decode (s) | Ref preprocess (s) | Post-process (s) | CPU MP4 (s) | Resident weights (GB) | Peak memory (GB) |
| ---- | ---- | ---- | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| t2va | 1344x768 | 124 | 5 | lossless | 150.06 | 139.9 | 2.85 | 49 | 0.83 | NA | NA | 6.76 | NA | 0.462 | 1.18 | 2.97 | 24.7 |
| t2va | 1344x768 | 362 | 15 | lossless | 786.24 | 761.36 | 15.53 | 49 | 1.20 | NA | NA | 15.87 | NA | 1.98 | 2.95 | 2.97 | 36.21 |
| t2va | 1344x768 | 124 | 5 | lossy | 108.1 | 95.85 | 1.95 | 49 | 0.9 | NA | NA | 6.73 | NA | 0.745 | 1.04 | 2.55 | 22.47 |
| t2va | 1344x768 | 362 | 15 | lossy | 552.51 | 526.04 | 10.73 | 49 | 1.28 | NA | NA | 16.07 | NA | 1.90 | 3.76 | 2.55 | 35.11 |
| ref2va | 1344x768 | 124 | 5 | lossless | 474.89 | 421.68 | 8.6 | 49 | 2.42 | 13.31 | 0.38 | 6.47 | 3.94 | 0.489 | 1.16 | 3.4 | 23.45 |
| ref2va | 1344x768 | 362 | 15 | lossless | 3414.18 | 3336.98 | 68.08 | 49 | 6.48 | 30.85 | 0.59 | 15.8 | 12.59 | 1.99 | 3.08 | 3.4 | 36.28 |
| ref2va | 1344x768 | 124 | 5 | lossy | 380.92 | 326.50 | 6.66 | 49 | 2.47 | 13.48 | 0.38 | 6.36 | 4.68 | 0.591 | 1.22 | 2.55 | 22.57 |
| ref2va | 1344x768 | 362 | 15 | lossy | 2332.04 | 2249.63 | 45.89 | 49 | 6.33 | 35.1 | 0.59 | 15.72 | 11.07 | 1.35 | 2.3 | 2.55 | 35.58 |
Keep `--ring 1` when using RainFusion: the `rf_v2` kernel ranks key blocks
over the whole sequence, so ring parallelism would split away the keys it
needs. Scale with `--usp` instead.
### Atlas 800I A3
## Memory and attention optimizations (A3)
| Task | Resolution | Frames | Duration (s) | Config | E2E (s) | DiT total (s) | DiT per-step (s) | DiT steps | Text encode (s) | Ref video encode (s) | Ref audio encode (s) | VAE decode (s) | Ref preprocess (s) | Post-process (s) | CPU MP4 (s) | Resident weights (GB) | Peak memory (GB) |
| ---- | ---- | ---- | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| t2va | 1344x768 | 124 | 5 | lossless | 114.19 | 105.87 | 2.16 | 49 | 0.48 | NA | NA | 6.56 | NA | 0.001 | 0.46 | 3.49 | 24.7 |
| t2va | 1344x768 | 362 | 15 | lossless | 599.29 | 580.38 | 11.84 | 49 | 1.83 | NA | NA | 14.95 | NA | 0.001 | 1.02 | 3.49 | 36.21 |
| t2va | 1344x768 | 124 | 5 | lossy | 84.82 | 77.47 | 1.58 | 49 | 0.57 | NA | NA | 6.23 | NA | 0.001 | 0.61 | 2.63 | 22.47 |
| t2va | 1344x768 | 362 | 15 | lossy | 401.43 | 383.56 | 7.82 | 49 | 1.72 | NA | NA | 14.83 | NA | 0.001 | 1.14 | 2.63 | 35.11 |
| ref2va | 1344x768 | 124 | 5 | lossless | 355.53 | 311.86 | 6.36 | 49 | 1.89 | 12.37 | 0.24 | 5.55 | 2.49 | 0.001 | 0.43 | 3.49 | 23.45 |
| ref2va | 1344x768 | 362 | 15 | lossless | 2339.56 | 2277.61 | 46.47 | 49 | 5.61 | 34.46 | 0.36 | 13.61 | 6.83 | 0.001 | 1.11 | 3.49 | 36.28 |
| ref2va | 1344x768 | 124 | 5 | lossy | 306.82 | 247.72 | 5.05 | 49 | 1.88 | 12.71 | 0.24 | 5.24 | 2.45 | 0.001 | 0.54 | 2.63 | 22.57 |
| ref2va | 1344x768 | 362 | 15 | lossy | 1695.26 | 1634.58 | 33.35 | 49 | 5.57 | 33.27 | 0.36 | 13.51 | 6.96 | 0.001 | 1.22 | 2.63 | 35.58 |
### Fitting 768P into 64 GB HBM: distributed layerwise offload or HSDP
Each Atlas 800I A3 NPU has 64 GB of HBM, which is not enough for the
combined MiniMax-H3 service at 768P. Enable **one** of the following at
server startup:
**Distributed layerwise offload** — DiT layers are offloaded with parameters
gathered across the parallel group instead of replicated per rank:
```bash
--enable-distributed-layerwise-offload
```
Measured peak NPU memory is about 45 GB per device for Ref2VA with a 13.88 s
input video generating a 15 s 768P (1344x768) output video. Host (CPU) memory
usage is high with this option.
**HSDP** — hybrid sharded data parallelism for the DiT parameters. The
multi-stream memory-reuse knob is mandatory with this flag:
```bash
export MULTI_STREAM_MEMORY_REUSE=2
```
```bash
--use-hsdp
```
Host memory usage is small, but large shapes may still OOM; HBM usage
optimizations for this path are ongoing.
### FLASH_ATTN backend with MindIE-SD
Keep `--diffusion-attention-backend FLASH_ATTN` and install MindIE-SD (see
[§ Environment](#environment)). On NPU this backend carries most of the
memory-reduction and performance work: a mask-free packed varlen path that
never materializes the quadratic `full_qk` padding mask, and K/V prefix
slicing, both driven by the packed `cu_seqlens` metadata emitted by the H3
transformer.
### Optional: LaserAttention fused kernel
For an additional attention speedup, select the Ascend LaserAttention fused
kernel before starting the server:
```bash
export MINDIE_SD_FA_TYPE="ascend_laser_attention"
```
This requires the FLASH_ATTN backend and MindIE-SD. H3 automatically applies
exact power-of-two input pre-scaling (`laser_input_scale=256`) so the
kernel's fp16 workspace cannot overflow on outlier activations. Measured
speedup numbers will be added here.
## HTTP API examples
The request format is identical to the GPU recipe; see
[MiniMax-H3.md § HTTP API examples](MiniMax-H3.md#http-api-examples).
Use the validated 768P shapes (e.g. `width=1344 height=768`) on NPU.
## Key parameters
Same as the GPU recipe; see
[MiniMax-H3.md § Key parameters](MiniMax-H3.md#key-parameters).
The validated resolution on NPU is 768P (e.g. 1344x768).
## Validated NPU evidence
Measured on an Atlas 800I A3 server (8x NPU) with CANN 9.0.1,
PyTorch 2.10.0+cpu, and torch_npu 2.10.0.post2, using the multi-NPU
configuration above:
| Workload | Configuration |
|----------|---------------|
| T2VA, 209 frames, 1344x768 | TE TP8, distributed layerwise offload, Ulysses 8, VPP8 tile, regional compile |
| Ref2VA (prompt + video), 124 frames, 1344x768 | TE TP8, distributed layerwise offload, Ulysses 8, VPP8 tile, regional compile |
These measurements describe the validated shapes rather than a general
throughput guarantee.
These numbers describe the validated shapes rather than a general throughput
guarantee.
## Known limitations
- Combined serving requires sibling `FL2VA` and `Ref2VA` directories, loads
both task-specific DiTs, and loads one copy of every shared component.
- Task serving is partitioned by `--task-type`: T2VA/FL2VA and Ref2VA load
different DiTs, so switching between them requires a restart with the other
partition.
- H3 currently executes one generation request per diffusion batch.
- The first regional-compile request is a warmup and should not be included
in steady-state performance measurements.
@@ -250,17 +380,19 @@ throughput guarantee.
the same HTTP request contract.
- VAE patch parallelism requires size 1 or the full DiT group size and
supports the H3 native `tile` mode only.
- RainFusion block-sparse attention and INT8 quantization are validated for
T2VA only; use the BF16 dense configuration for FL2VA and Ref2VA.
- Online INT8 quantization can be combined with distributed layerwise offload
while AllGather is enabled. Keep AllGather enabled for SP>1; use
`--dlo-no-use-allgather` only when collective communication is unavailable.
- The lossy configuration (RainFusion + INT8) is validated for T2VA and
Ref2VA; use the lossless configuration for FL2VA or re-validate first.
## Additional resources
- [MiniMax-H3.md](MiniMax-H3.md) — full GPU guide
- [Attention backends § RAINFUSION_ATTN](../../docs/user_guide/diffusion/attention_backends.md#rainfusion_attn-backend-and-block-sparse-video-attention)
— RainFusion knobs and tuning
- [Int8 quantization](../../docs/user_guide/quantization/int8.md)
- [MiniMax-H3-NPU-950PR.md](MiniMax-H3-NPU-950PR.md) — four-card
Ascend 950PR / 950DT guide
- [NPU installation](../../docs/getting_started/installation/npu.md)
- [RainFusion attention](../../docs/user_guide/diffusion/attention_backends/rainfusion.md)
— block-sparse knobs and tuning
- [Online quantization](../../docs/user_guide/quantization/online.md)
— INT8 / MXFP8 / MXFP4
- [Cache-DiT guide](../../docs/user_guide/diffusion/cache_acceleration/cache_dit.md)
- [Supported models](../../docs/models/supported_models.md)
- [Video API](../../docs/serving/videos_api.md)