[Model][MiniMax-H3] Multi-GPU serving fixes and layer-wise offload component selection (#7047)
Signed-off-by: heyuanliu-intel <heyuan.liu@intel.com>
This commit is contained in:
1 parent
2f1a18e69f
commit
40f8e94400
21 files changed
+1857
-60
No files matched your search
@@ -34,5 +34,34 @@ that provide their own `setup_compile()` implementation manage their compilation
|
||||
policy independently. Compilation is lazy, so backend or graph errors can first
|
||||
surface on the initial request.
|
||||
|
||||
## Pinning one packed shape (MiniMax-H3)
|
||||
|
||||
Compiled regions are keyed by input shape. MiniMax-H3 packs a request into a
|
||||
row count that follows the request -- a longer prompt, more frames, a larger
|
||||
frame -- and rounds it up to the next 64-row boundary, so two requests can land
|
||||
on two different packed lengths.
|
||||
|
||||
Under the default `diffusion_compile_dynamic: true` a new packed length is
|
||||
absorbed by the dynamic shape, so it is not by itself a recompilation. It still
|
||||
hands the backend a shape it has not autotuned, which is the risk this knob
|
||||
exists to remove: a fixed-shape deployment (`--no-diffusion-compile-dynamic`)
|
||||
recompiles the transformer blocks per shape, and a first-seen large shape can
|
||||
push autotuning into its worst-case memory use.
|
||||
|
||||
A request pins the length with `extra_args["pad_seq_len"]`:
|
||||
|
||||
```python
|
||||
sampling_params = SamplingParams(extra_args={"pad_seq_len": 54080})
|
||||
```
|
||||
|
||||
The value must be a positive multiple of 64 and must cover the rows the request
|
||||
actually uses; the packed sequence is then padded to it instead of to the next
|
||||
64-row boundary. Pick a bucket that covers every request dimension the
|
||||
deployment accepts -- prompt length, frame count, frame size and reference
|
||||
blocks all change the used row count. The server logs the effective length as
|
||||
`MiniMax H3 packed sequence: ... pad_seq_len=... used=... seq_len=...` whenever
|
||||
a request pins it. The padding rows are masked out, so their cost is the
|
||||
attention and feed-forward work on those rows.
|
||||
|
||||
Use `--enforce-eager` to disable the model runner's generic compile setup.
|
||||
Pipelines that compile internally define their own eager-mode behavior.
|
||||
Reference in new issue
Block a user