[Feature] Support DiT Layerwise (Blockwise) CPU Offloading (#858)
Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
14 files changed
+613
-41
No files matched your search
@@ -288,6 +288,12 @@ class OmniDiffusionConfig:
|
||||
# - Text encoders run on GPU while DiT is on CPU
|
||||
# - DiT runs on GPU while encoders are on CPU
|
||||
enable_cpu_offload: bool = False
|
||||
|
||||
# Layer-wise offloading (block-level offloading) parameters
|
||||
enable_layerwise_offload: bool = False
|
||||
# Number of transformer blocks ready for computation to keep on GPU
|
||||
layerwise_num_gpu_layers: int = 1
|
||||
|
||||
use_fsdp_inference: bool = False
|
||||
pin_cpu_memory: bool = True # Use pinned memory for faster transfers when offloading
|
||||
|
||||
|
||||
Reference in new issue
Block a user