[Model][MiniMax-H3] Multi-GPU serving fixes and layer-wise offload component selection (#7047)

Signed-off-by: heyuanliu-intel <heyuan.liu@intel.com>
This commit is contained in:
heyuanliu-intel authored and GitHub committed 2026-09-14 10:02:30 -07:00
1 parent 2f1a18e69f
commit 40f8e94400
21 files changed
+1857 -60

No files matched your search

@@ -34,5 +34,34 @@ that provide their own `setup_compile()` implementation manage their compilation
policy independently. Compilation is lazy, so backend or graph errors can first
surface on the initial request.
## Pinning one packed shape (MiniMax-H3)
Compiled regions are keyed by input shape. MiniMax-H3 packs a request into a
row count that follows the request -- a longer prompt, more frames, a larger
frame -- and rounds it up to the next 64-row boundary, so two requests can land
on two different packed lengths.
Under the default `diffusion_compile_dynamic: true` a new packed length is
absorbed by the dynamic shape, so it is not by itself a recompilation. It still
hands the backend a shape it has not autotuned, which is the risk this knob
exists to remove: a fixed-shape deployment (`--no-diffusion-compile-dynamic`)
recompiles the transformer blocks per shape, and a first-seen large shape can
push autotuning into its worst-case memory use.
A request pins the length with `extra_args["pad_seq_len"]`:
```python
sampling_params = SamplingParams(extra_args={"pad_seq_len": 54080})
```
The value must be a positive multiple of 64 and must cover the rows the request
actually uses; the packed sequence is then padded to it instead of to the next
64-row boundary. Pick a bucket that covers every request dimension the
deployment accepts -- prompt length, frame count, frame size and reference
blocks all change the used row count. The server logs the effective length as
`MiniMax H3 packed sequence: ... pad_seq_len=... used=... seq_len=...` whenever
a request pins it. The padding rows are masked out, so their cost is the
attention and feed-forward work on those rows.
Use `--enforce-eager` to disable the model runner's generic compile setup.
Pipelines that compile internally define their own eager-mode behavior.