0 Fork 0

Files

T

hsliuustc0106 e1c4b6153a [Model] Make the dummy-run recipe feature-neutral

Per review, the warmup recipe is a property of the model's request
geometry, not of distributed layerwise offload: rename
dlo_dummy_run_recipe to dummy_run_recipe and drop the DLO gate in
_make_dummy_request. A model whose geometry needs recipe-specific sampling
arguments warms up at startup whatever features are enabled — under DLO
the generation also primes the component cache-retention budget; without
it, the same warmup covers any other first-request cost. Validated both
ways on L20X: DLO boot unchanged (releases stay in the startup window) and
a non-DLO MiniMax-H3 boot now runs the warmup and serves normally.

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>

2026-08-26 15:48:07 +08:00

cpu_offload.md

[Docs] Split diffusion attention and CPU offload guides (#6075 )

2026-08-15 14:41:09 +08:00

distributed_layerwise_offload.md

[Model] Make the dummy-run recipe feature-neutral

2026-08-26 15:48:07 +08:00

layerwise_offload.md

[Docs] Split diffusion attention and CPU offload guides (#6075 )

2026-08-15 14:41:09 +08:00

README.md

[Docs] Split diffusion attention and CPU offload guides (#6075 )

2026-08-15 14:41:09 +08:00

README.md

CPU Offloading

This document defines the shared architecture for diffusion CPU offloading. The three strategies have separate user and design pages:

Strategy	User guide	Design contract
Model-level	Guide	Design
Layerwise	Guide	Design
Distributed layerwise	Guide	Design

Strategy selection

OffloadConfig.from_od_config() converts diffusion configuration into one OffloadStrategy. If callers set multiple flags, selection is deterministic:

DISTRIBUTED_LAYERWISE;
LAYERWISE;
MODEL_LEVEL; and
NONE.

The factory logs when a higher-priority strategy disables another. It derives parallel and HSDP state from DiffusionParallelConfig; callers do not provide a separate offload group size.

get_offload_backend() then validates platform offload support, resolves the device, and creates exactly one backend. Returning None means offloading is disabled or unsupported and must not leave partially installed hooks.

Shared lifecycle

Every backend implements OffloadBackend:

enable(pipeline) discovers modules, establishes initial residency, and installs hooks;
disable() removes owned hooks and resources; and
is_enabled() reports lifecycle state.

disable() does not promise to restore the pipeline's original device placement. The caller owns any subsequent rematerialization.

Hooks are registered through HookRegistry and ModelHook; offload backends must use distinct hook names and remove only hooks they own.

Discovery and topology

Pipeline component discovery prefers SupportsComponentDiscovery declarations:

_dit_modules;
_encoder_modules;
_vae_modules; and
_resident_modules.

Dotted paths are supported. Legacy pipelines may use the fallback scan of well-known attribute names, but new integrations should declare components explicitly.

Ordinary layerwise offload obtains topology from declared _layerwise_offload_blocks_attrs (with a deprecated singular-name compatibility path). Distributed layerwise offload may instead use a pipeline OffloadPlan, then falls back to declared block attributes and its documented compatibility discovery. Discovery metadata describes structure only; the backend remains responsible for transfer, synchronization, and storage ownership.

Cross-strategy invariants

At most one strategy owns offload hooks for a pipeline.
A parameter has one authoritative host representation while offloaded.
Device storage is not freed until dependent compute or transfer work is complete.
Non-persistent and model-specific buffers remain correct after movement or checkpoint rematerialization.
Unsupported parallel or loading combinations fail before hooks mutate the model.
Platform streams, events, synchronization, and cache management go through the vLLM-Omni platform abstraction.

The diffusion Offloader module design describes how these feature contracts fit into the larger diffusion runtime.