[Performance] Add per-component DLO offload policy

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
hsliuustc0106 committed 2026-08-26 07:23:36 +08:00
1 parent 780e8d40aa
commit 7db3a0cd5e
15 files changed
+324 -21

No files matched your search

@@ -29,6 +29,7 @@ Legend: ✅ supported, ⚠️ compatibility path or limited validation, ❌ unsu
| **Other online quantization methods** | ❌ Rejected until runtime packing and scale layouts are validated. | ⚠️ Allowed through the ordinary loader; validation is method-specific. |
| **Model-level or standard layerwise CPU offload** | ❌ Disabled because DLO takes priority. | ❌ Disabled because DLO takes priority. |
| **Resident leading layers** | ❌ Rejected. | ✅ Requires eligible resident paths in the model's `OffloadPlan`. |
| **Auxiliary component policy** | ✅ Encoders/VAEs may remain resident. | ✅ Encoders/VAEs may remain resident. |
See [Parallelism compatibility](#parallelism-compatibility) and
[Request and loading constraints](#request-and-loading-constraints) for the
@@ -419,6 +420,22 @@ failure.
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
configuration.
### Auxiliary component policy
`dlo_offload_components` maps discovered encoder/VAE paths to booleans. A
component set to `False` is materialized on the target device once and is not
entered through the model's manual offload lifecycle. A `True` value permits
the existing `OffloadPlan.on_demand_component_paths` and encoder block-hook
behavior; it does not add support where the model declares none. The optional
`default` key handles unmatched auxiliary components, and an empty map
preserves the current policy.
The policy is resolved after component discovery and before DLO mutates DiT
storage. Unknown names, including DiT paths, are rejected. DLO continues to
own DiT streaming regardless of this map, which avoids an invalid state where
the loader has skipped ordinary DiT materialization but the backend is asked
to keep that DiT resident.
## Request and loading constraints
AllGather DP multi-concurrency requires:
@@ -430,6 +447,15 @@ AllGather DP multi-concurrency requires:
The no-AllGather path does not impose these DLO-specific synchronized-wave
requirements.
MiniMax-H3 currently builds one global encoder TP group and broadcasts the
resulting conditioning from global rank 0. Therefore an encoder TP group that
spans multiple DP replicas is not valid for independent-prompt AllGather
waves. Exact-recipe DP2/TP2 and DP2/SP2 experiments with encoder TP4 completed
the collective schedule, but two distinct prompts with the same seed produced
byte-identical video and audio in every measured wave. Enabling or disabling
encoder offload produced paired-identical results, so this is a pre-existing
model topology limitation rather than component-policy behavior.
Direct checkpoint mmap can back either transfer path. It is currently limited
to proven TP1, non-HSDP, non-online-quantized layouts. Other layouts use the
ordinary loader. Per-tensor online FP8 linears can use DLO AllGather after the
@@ -494,6 +520,9 @@ AllGather, because DP replicas did not share checkpoint-backed runtime
weights. This is a functional and memory smoke test, not a production-quality
performance or output-quality benchmark.
These B300 DP waves intentionally reused the same prompt and seed, so they do
not establish independent-prompt isolation across DP replicas.
### Host-memory measurement
A two-worker MiniMax-H3 FL2VA measurement on one L20X node compared the
@@ -36,7 +36,8 @@ vllm serve /path/to/model --omni \
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4 \
--dlo-no-use-allgather
--dlo-no-use-allgather \
--dlo-offload-components '{"text_encoder": false}'
# Sequence parallel deployment
vllm serve /path/to/model --omni \
@@ -51,6 +52,7 @@ omni = Omni(
model="/path/to/model",
enable_distributed_layerwise_offload=True,
dlo_use_allgather=True,
dlo_offload_components={"text_encoder": False},
)
```
@@ -63,6 +65,7 @@ omni = Omni(
| `--dlo-use-allgather` | Shard host weights and reconstruct with AllGather | `true` |
| `--dlo-no-use-allgather` | Stream complete rank-local blocks without a DLO weight collective | `false` |
| `--dlo-resident-layers N` | Keep N leading main-DiT blocks on device; requires no-AllGather and model-declared resident paths | `0` |
| `--dlo-offload-components JSON` | Permit or disable offload for discovered encoder/VAE components | `{}` |
| `--host-weight-runtime-mode {disabled,preferred,required}` | HWR policy: no interaction, populate on a miss, or require an exact hit | `disabled` |
| `--host-weight-runtime-root PATH` | Writable node-local HWR store shared by workers in one storage domain; required for `preferred` and `required` | unset |
| `--dlo-host-registration-limit-gib N` | Optional per-worker ceiling for registering an HWR mmap; zero adds no ceiling | `0` |
@@ -244,6 +247,25 @@ class MyPipeline(nn.Module):
When no plan exists, discovery falls back to
`_layerwise_offload_blocks_attrs` and then heuristic attribute lookup.
## Auxiliary component policy
`dlo_offload_components` follows the same component-map style as diffusion
quantization. Keys are discovered encoder or VAE paths, values are booleans,
and `default` is the optional fallback:
```python
dlo_offload_components = {
"text_encoder": False,
"default": True,
}
```
`True` permits the model's existing DLO behavior for that component. `False`
keeps the whole component resident, including its child blocks and non-block
modules. An empty map preserves current behavior. Unknown keys are rejected;
the map does not control DiT blocks because enabling DLO gives the backend
ownership of DiT streaming.
## Data-parallel concurrency
With `data_parallel_size > 1` and AllGather enabled, the scheduler can process
@@ -268,6 +290,13 @@ must enter each collective.
are validated.
- Resident leading layers require `--dlo-no-use-allgather` and a model
`OffloadPlan` that declares eligible `resident_dit_paths`.
- Component policy applies only to discovered auxiliary encoders and VAEs;
it cannot disable DLO for a DiT.
- MiniMax-H3's encoder TP group is global rather than DP-replica-local. An
encoder TP size that spans multiple DP replicas (for example TP4 with
DP2/TP2 or DP2/SP2 on four ranks) is not supported for independent-prompt
AllGather waves: the current pipeline broadcasts rank 0's conditioning to
both replicas. This limitation is independent of the component policy.
- DP concurrency requires an explicit, identical inference-step count.
Sharing quantized or otherwise unvalidated transformed layouts through a