diff --git a/docs/.nav.yml b/docs/.nav.yml index 68aa944eb..30ccafc25 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -62,9 +62,9 @@ nav: - Feature Compatibility: user_guide/feature_compatibility.md - CPU Offloading: - Overview: user_guide/diffusion/cpu_offload.md - - Model-Level: user_guide/diffusion/cpu_offload/model_level.md - - Layerwise: user_guide/diffusion/cpu_offload/layerwise.md - - Distributed Layerwise: user_guide/diffusion/cpu_offload/distributed_layerwise.md + - Model-Level: user_guide/diffusion/offloader/cpu_offload.md + - Layerwise: user_guide/diffusion/offloader/layerwise_offload.md + - Distributed Layerwise: user_guide/diffusion/offloader/distributed_layerwise_offload.md - Cache Acceleration: - TeaCache: user_guide/diffusion/cache_acceleration/teacache.md - Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md diff --git a/docs/design/feature/cpu_offloading.md b/docs/design/feature/cpu_offloading.md index a1a7124b2..c37b705d2 100644 --- a/docs/design/feature/cpu_offloading.md +++ b/docs/design/feature/cpu_offloading.md @@ -5,9 +5,9 @@ The three strategies have separate user and design pages: | Strategy | User guide | Design contract | | --- | --- | --- | -| Model-level | [Guide](../../user_guide/diffusion/cpu_offload/model_level.md) | [Design](model_level_offload.md) | -| Layerwise | [Guide](../../user_guide/diffusion/cpu_offload/layerwise.md) | [Design](layerwise_offload.md) | -| Distributed layerwise | [Guide](../../user_guide/diffusion/cpu_offload/distributed_layerwise.md) | [Design](distributed_layerwise_offload.md) | +| Model-level | [Guide](../../user_guide/diffusion/offloader/cpu_offload.md) | [Design](model_level_offload.md) | +| Layerwise | [Guide](../../user_guide/diffusion/offloader/layerwise_offload.md) | [Design](layerwise_offload.md) | +| Distributed layerwise | [Guide](../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) | [Design](distributed_layerwise_offload.md) | ## Strategy selection diff --git a/docs/design/feature/distributed_layerwise_offload.md b/docs/design/feature/distributed_layerwise_offload.md index 672b3c2e6..0d260ca25 100644 --- a/docs/design/feature/distributed_layerwise_offload.md +++ b/docs/design/feature/distributed_layerwise_offload.md @@ -7,7 +7,7 @@ either shard those host-side weights across an existing parallel group or keep the standard loader's rank-local weights and avoid an additional collective. For user-facing commands, see the -[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload/distributed_layerwise.md) +[distributed layerwise offloading guide](../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md). ## Status diff --git a/docs/design/feature/layerwise_offload.md b/docs/design/feature/layerwise_offload.md index 30b52ee62..94e9ff1b9 100644 --- a/docs/design/feature/layerwise_offload.md +++ b/docs/design/feature/layerwise_offload.md @@ -2,7 +2,7 @@ Layerwise offload streams transformer blocks through one accelerator. For commands and model support, see the -[layerwise user guide](../../user_guide/diffusion/cpu_offload/layerwise.md). +[layerwise user guide](../../user_guide/diffusion/offloader/layerwise_offload.md). ## Hook ring diff --git a/docs/design/feature/model_level_offload.md b/docs/design/feature/model_level_offload.md index fd213b3aa..b7e80d04d 100644 --- a/docs/design/feature/model_level_offload.md +++ b/docs/design/feature/model_level_offload.md @@ -2,7 +2,7 @@ Model-level offload enforces mutual exclusion between pipeline component groups. For commands and model support, see the -[model-level user guide](../../user_guide/diffusion/cpu_offload/model_level.md). +[model-level user guide](../../user_guide/diffusion/offloader/cpu_offload.md). ## Generic hook path diff --git a/docs/user_guide/diffusion/cpu_offload.md b/docs/user_guide/diffusion/cpu_offload.md index 0bb798735..c1396f12b 100644 --- a/docs/user_guide/diffusion/cpu_offload.md +++ b/docs/user_guide/diffusion/cpu_offload.md @@ -11,9 +11,9 @@ For the shared factory, discovery, and lifecycle contract, see the | Strategy | Device residency | Parallel scope | Guide | | --- | --- | --- | --- | -| Model-level (sequential) | One pipeline component group at a time | Single device | [Model-Level Offloading](cpu_offload/model_level.md) | -| Layerwise (blockwise) | One transformer block, with next-block prefetch | Single device | [Layerwise Offloading](cpu_offload/layerwise.md) | -| Distributed layerwise | Fixed two-block device buffer; optional host-weight sharding and AllGather | Multiple GPU/NPU ranks | [Distributed Layerwise Offloading](cpu_offload/distributed_layerwise.md) | +| Model-level (sequential) | One pipeline component group at a time | Single device | [Model-Level Offloading](offloader/cpu_offload.md) | +| Layerwise (blockwise) | One transformer block, with next-block prefetch | Single device | [Layerwise Offloading](offloader/layerwise_offload.md) | +| Distributed layerwise | Fixed two-block device buffer; optional host-weight sharding and AllGather | Multiple GPU/NPU ranks | [Distributed Layerwise Offloading](offloader/distributed_layerwise_offload.md) | All strategies use pinned host memory for faster transfers where applicable. Configuration priority is: @@ -27,12 +27,12 @@ higher-priority strategy is selected. ## Quick selection -- Use [model-level offloading](cpu_offload/model_level.md) when swapping whole +- Use [model-level offloading](offloader/cpu_offload.md) when swapping whole encoders and DiTs is enough to fit and phase-boundary transfers are acceptable. -- Use [layerwise offloading](cpu_offload/layerwise.md) for compute-heavy video +- Use [layerwise offloading](offloader/layerwise_offload.md) for compute-heavy video DiTs where block transfers can overlap computation. -- Use [distributed layerwise offloading](cpu_offload/distributed_layerwise.md) +- Use [distributed layerwise offloading](offloader/distributed_layerwise_offload.md) when a multi-rank deployment also needs bounded device residency and, optionally, sharded host weights. @@ -64,12 +64,12 @@ These headings preserve links to sections that moved into dedicated guides. ## Model-level (Sequential) Offloading -See [Model-Level Offloading](cpu_offload/model_level.md). +See [Model-Level Offloading](offloader/cpu_offload.md). ## Layerwise (Blockwise) Offloading -See [Layerwise Offloading](cpu_offload/layerwise.md). +See [Layerwise Offloading](offloader/layerwise_offload.md). ## Distributed Layerwise Offloading -See [Distributed Layerwise Offloading](cpu_offload/distributed_layerwise.md). +See [Distributed Layerwise Offloading](offloader/distributed_layerwise_offload.md). diff --git a/docs/user_guide/diffusion/cpu_offload/model_level.md b/docs/user_guide/diffusion/offloader/cpu_offload.md similarity index 100% rename from docs/user_guide/diffusion/cpu_offload/model_level.md rename to docs/user_guide/diffusion/offloader/cpu_offload.md diff --git a/docs/user_guide/diffusion/cpu_offload/distributed_layerwise.md b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md similarity index 100% rename from docs/user_guide/diffusion/cpu_offload/distributed_layerwise.md rename to docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md diff --git a/docs/user_guide/diffusion/cpu_offload/layerwise.md b/docs/user_guide/diffusion/offloader/layerwise_offload.md similarity index 100% rename from docs/user_guide/diffusion/cpu_offload/layerwise.md rename to docs/user_guide/diffusion/offloader/layerwise_offload.md diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 67ace0940..11c6366b7 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -217,7 +217,7 @@ The following tables show which models support each feature: 3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible. 4. The CPU Offloading (Layerwise) row describes local layerwise offload. Multi-device Distributed Layerwise Offload has a separate topology and - compatibility matrix in the [Distributed Layerwise Offloading guide](diffusion/cpu_offload/distributed_layerwise.md). + compatibility matrix in the [Distributed Layerwise Offloading guide](diffusion/offloader/distributed_layerwise_offload.md). 5. The compatibility matrix uses FP8 as the representative quantization method. 6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).