Extend Cosmos3 CPU offloading to component and layerwise paths (#4695)

This commit is contained in:
Rahul Steiger authored and GitHub committed 2026-07-13 09:48:53 +00:00
1 parent 98229b08ba
commit 66b9ad287a
11 files changed
+653 -160

No files matched your search

@@ -82,6 +82,33 @@ mutual exclusion: when one group runs, the other moves to CPU.
- Support single GPU only for now
### Component offloading for split models (e.g. Cosmos3)
Some models split their transformer into mutually-exclusive *components* that run
in different phases of a single forward pass rather than as separate pipeline
components -- e.g. Cosmos3's understanding (reasoner) component runs once per
generation while the generation (generator) component runs every denoising step.
Such models have no separate text encoder to swap against, so the transformer
owns a small model-local offload path and wraps each phase with
`with self._offload_context(name):`
```python
class Cosmos3VFMTransformer(nn.Module):
def forward(self, ...):
with self._offload_context("reasoner"):
... # understanding pass, runs once
with self._offload_context("generator"):
... # denoising pass, runs every step
```
Model-level offloading then keeps exactly one component GPU-resident at a time
(the other on CPU), reusing the same `SequentialOffloadHook` `.to()` movers. The
pipeline opts in by exposing `enable_omni_model_cpu_offload` (which drives the
transformer's `enable_model_cpu_offload` and pins the VAE). Layerwise offloading
works for these models too -- each component declares its own block container via
`_layerwise_offload_blocks_attrs`.
## Layerwise (Blockwise) Offloading
### How It Works
@@ -145,9 +172,9 @@ class Flux2Transformer2DModel(nn.Module):
```
### Limitations
- Cold start latency increases because of
1) components are loaded to CPU first at the very first during initialization,
2) weight consolidation and pinning
- Cold start latency increases because offloaded components must be moved to CPU
during setup; layerwise offload may add extra weight consolidation and pinning
work.
- Performance depends on compute cost and H2D bandwidth as well
- Support single GPU only for now
@@ -178,11 +205,18 @@ Both strategies use vLLM-Omni's hook registry system (`HookRegistry` and `ModelH
```
OffloadBackend (base class)
├── ModelLevelOffloadBackend → uses SequentialOffloadHook
├── ModelLevelOffloadBackend → uses SequentialOffloadHook (.to() swap)
│ (delegates to a pipeline's enable_omni_model_cpu_offload
│ for split models like Cosmos3)
└── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
```
Factory function `get_offload_backend()` selects the appropriate backend based on configuration.
Factory function `get_offload_backend()` selects the appropriate backend based on
configuration.
For split models, `ModelLevelOffloadBackend.enable()` detects a pipeline's
`enable_omni_model_cpu_offload` hook and delegates to it; Cosmos3 then swaps its
reasoner/generator components inside the model forward pass.
## Supported Models
+2 -2
View File
@@ -105,7 +105,7 @@ The following tables show which models support each feature:
> Notes:
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only.
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload_diffusion.md).
> 2. The **💾Quantization** column is collapsed for readability. See [Quantization Overview](quantization/overview.md) for per-method and per-model support details.
### ImageGen
@@ -144,7 +144,7 @@ The following tables show which models support each feature:
> Notes:
> 1. Nextstep_1(T2I) does not support cache acceleration methods such as TeaCache or Cache-DiT.
> 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary.
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload is not supported; use layerwise offload.
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload.
> 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`.
### VideoGen