Gate persistent DiT staging on decode-graph models
ModelLevelOffloadBackend passed persistent_dit_staging=(device is CUDA) for every model, so Qwen-Image 2.0/FLUX/WAN-style sequential offload kept DiT device storage alive across swaps instead of freeing VRAM. Only retain staging when a DiT actually captures weight pointers (enable_cuda_graph_decode); other models keep plain move semantics. Add tests pinning both behaviors.
This commit is contained in:
1 parent
be8a72f0a6
commit
0c82bb1315
2 files changed
+56
-2
No files matched your search
@@ -415,8 +415,14 @@ class ModelLevelOffloadBackend(OffloadBackend):
|
||||
offload_dit_modules=selected_dits,
|
||||
offload_encoder_modules=selected_encoders,
|
||||
# Fixed DiT staging keeps decode-graph weight pointers valid;
|
||||
# other platforms keep plain move semantics.
|
||||
persistent_dit_staging=self.device.type == "cuda",
|
||||
# only models that capture weight pointers (Qwen-Image-2.1's
|
||||
# CUDA-graph decode) pay for it — every other model keeps
|
||||
# plain move semantics so offloading actually frees DiT VRAM
|
||||
# while the encoders run.
|
||||
persistent_dit_staging=(
|
||||
self.device.type == "cuda"
|
||||
and any(getattr(dit, "enable_cuda_graph_decode", False) for dit in dits)
|
||||
),
|
||||
)
|
||||
except BaseException:
|
||||
try:
|
||||
|
||||
Reference in new issue
Block a user