diff --git a/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md b/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md new file mode 100644 index 000000000..e897c3de4 --- /dev/null +++ b/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md @@ -0,0 +1,149 @@ +# MiniMax-H3 on DGX Spark (GB10) + +This recipe uses online FP8 weight quantization, tiled VAE decode, and a single +resident partition. GB10 is a unified-memory platform, so unlike the discrete-GPU +recipes it uses **no offload of any kind** — see the capacity note below. + +## Capacity requirements + +| Resource | DGX Spark (GB10) | +| --- | ---: | +| Unified memory | 128 GB LPDDR5X (~121 GiB usable) | +| GPUs | 1 | +| Checkpoint storage | 135 GiB per partition | +| Measured peak allocator high-water (FP8) | 97.7 GiB | + +`FL2VA` and `Ref2VA` are separate 135 GiB checkpoint partitions. Start one server +at a time. + +On GB10 the CPU and GPU share one physical memory pool, which changes the capacity +math relative to the RTX and datacenter recipes: + +- **Do not use `--enable-distributed-layerwise-offload`.** DLO stages rank-local + weights into pinned host memory, which on GB10 is the same pool the weights are + already in. Peak usage roughly doubles and the process is killed by the OOM + killer (`Exit code: -9`) shortly after `Enabling offloader backend`. +- **Do not use `--enable-cpu-offload`.** Moving weights from "VRAM" to "host RAM" + frees nothing here. +- **`--quantization fp8` is mandatory.** A BF16 partition is 135 GiB and does not + fit in 121 GiB. Online FP8 quantizes the DiT only (62 GiB to ~31 GiB); the + Qwen3-VL text encoder and both VAEs stay BF16. + +With ~97.7 GiB of the pool held by the allocator at peak, there is roughly 23 GiB +of headroom for the OS, page cache, and any other process on the box. Treat this +platform as single-tenant while a request is in flight. + +## One GB10: 960x576, 8 seconds + +```bash +CUDA_VISIBLE_DEVICES=0 \ +FLASHINFER_DISABLE_VERSION_CHECK=1 \ +VLLM_WORKER_MULTIPROC_METHOD=spawn \ +VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ +vllm serve /path/to/MiniMax-H3/FL2VA \ + --omni --trust-remote-code --host 0.0.0.0 --port 8000 \ + --init-timeout 3600 \ + --num-gpus 1 --tensor-parallel-size 1 --text-encoder-tp-size 1 \ + --usp 1 --ring 1 --vae-patch-parallel-size 1 \ + --vae-parallel-mode tile --vae-use-tiling \ + --quantization fp8 --enforce-eager \ + --diffusion-attention-backend CUDNN_ATTN +``` + +Wait for `Application startup complete`, then run a short smoke test before +committing to a full 50-step request. + +`t2va` requires an explicit `aspect_ratio` even when `width` and `height` are +supplied; omitting it fails the request with +`OmniClientError: t2va requires an explicit aspect_ratio`. + +```bash +curl --fail-with-body -sS -X POST http://127.0.0.1:8000/v1/videos/sync \ + -F 'prompt=At night, three cats march into a bedroom playing tiny brass instruments, then abruptly file out, with synchronized room ambience.' \ + -F 'width=960' \ + -F 'height=576' \ + -F 'aspect_ratio=16:9' \ + -F 'fps=24' \ + -F 'num_inference_steps=10' \ + -F 'flow_shift=12' \ + -F 'seed=1101' \ + -F 'extra_params={"task":"t2va","duration":4.0,"audio_flow_shift":3.0}' \ + -o h3-gb10-smoke.mp4 +``` + +The full-quality request uses 50 steps: + +```bash +curl --fail-with-body -sS -X POST http://127.0.0.1:8000/v1/videos/sync \ + -F 'prompt=At night, three cats march into a bedroom playing tiny brass instruments, then abruptly file out, with synchronized room ambience.' \ + -F 'width=960' \ + -F 'height=576' \ + -F 'aspect_ratio=16:9' \ + -F 'fps=24' \ + -F 'num_inference_steps=50' \ + -F 'flow_shift=12' \ + -F 'seed=1101' \ + -F 'extra_params={"task":"t2va","duration":8.0,"audio_flow_shift":3.0}' \ + -o h3-gb10-t2va.mp4 + +ffprobe -v error -show_entries \ + stream=index,codec_name,width,height,r_frame_rate,sample_rate,channels \ + -of json h3-gb10-t2va.mp4 +``` + +Both requests return `200 OK` with an MP4 body: H.264 video at the requested +geometry plus a 32 kHz stereo AAC track. + +## Measured latency + +Single GB10, FP8, eager, cuDNN attention, tiled VAE, one request at a time: + +| Workload | Steps | Text encode | Denoise | Per step | VAE decode | MP4 mux | Client E2E | Peak memory | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| T2VA, 4 s, smoke shape | 10 | — | — | 8.68 s | — | 0.49 s | 95.9 s | — | +| T2VA, 8 s, 960x576 | 50 | 0.25 s | 2088 s | 42.61 s | 70 s | 4.84 s | 2169.4 s (36 min 9 s) | 97.7 GiB | + +How these were measured: + +- Stage times come from `--enable-diffusion-pipeline-profiler`, read from the + `X-Stage-Durations` response header of `/v1/videos/sync`. +- Peak memory is the `X-Peak-Memory-MB` header, which reports the CUDA allocator + high-water mark (`torch.cuda.max_memory_reserved`) for the request. It excludes + the CUDA context and non-PyTorch allocations, so true pool usage is somewhat + higher. `nvidia-smi --query-gpu=memory.used` is not usable on GB10 — unified + memory has no discrete VRAM to report and the field returns `[N/A]`. +- Text encode and VAE decode were measured on a separate 10-step run at the same + 960x576 / 8 s geometry. Both are independent of step count, so they transfer to + the 50-step row unchanged. +- Stage times do not sum exactly to end-to-end. Queueing, result transfer, and + engine bookkeeping sit outside the profiled stages. + +Denoise dominates at 96% of end-to-end, so step count and geometry are the only +knobs that materially change the wall clock on this platform. + +Repeated identical 10-step requests varied by up to 23% in denoise time +(397 s to 490 s) on a machine that had already been under sustained load. Treat +these as single-run measurements on one machine, not a throughput guarantee, and +expect the slower end of that range for back-to-back jobs. + +Set `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` well above the expected request time — the +default 1800 s is shorter than a 50-step run on this platform. + +## Known limitations + +- One GPU means no Ulysses, ring, TP, or VAE patch parallelism. Pass + `--usp 1 --ring 1 --vae-patch-parallel-size 1` explicitly if a profile or + script would otherwise supply higher degrees. +- `--enforce-eager` is used because regional compile adds startup time without a + parallelism win on a single rank. +- `t2va` rejects requests without an explicit `aspect_ratio`. When `width` and + `height` are both supplied they take precedence and `aspect_ratio` only has to + name one of the supported ratios. +- Sustained load degrades throughput. See the variance note above. +- Only one partition is resident. For Ref2VA, stop the FL2VA server and restart + the same command with `/path/to/MiniMax-H3/Ref2VA`. Ref2VA reference video + count and prompt length increase activation memory; begin with one request at + a time and keep the short edge at 576. +- Online FP8 is incompatible with layerwise offload — the offload path produces a + weight stride the Cutlass FP8 kernel rejects. This is not a practical + restriction here since offload is unusable on GB10 anyway.