diff --git a/docs/.nav.yml b/docs/.nav.yml index 30ccafc25..46651daf9 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -138,10 +138,10 @@ nav: - Backend Selection: design/feature/attention_backend_selection.md - Skip-Softmax: design/feature/skip_softmax.md - CPU Offloading: - - Overview: design/feature/cpu_offloading.md - - Model-Level: design/feature/model_level_offload.md - - Layerwise: design/feature/layerwise_offload.md - - Distributed Layerwise: design/feature/distributed_layerwise_offload.md + - Overview: design/feature/offloader/README.md + - Model-Level: design/feature/offloader/cpu_offload.md + - Layerwise: design/feature/offloader/layerwise_offload.md + - Distributed Layerwise: design/feature/offloader/distributed_layerwise_offload.md - Cache-DiT: design/feature/cache_dit.md - TeaCache: design/feature/teacache.md - Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md diff --git a/docs/design/architecture_overview.md b/docs/design/architecture_overview.md index aa6f72770..4ae1b495e 100644 --- a/docs/design/architecture_overview.md +++ b/docs/design/architecture_overview.md @@ -406,7 +406,7 @@ the source for configuration, compatibility, and implementation details. stage contract. * **Quantization and memory efficiency:** [Quantization](feature/quantization.md) resolves per-pipeline or per-component quantization configurations, while - [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md) + [Distributed Layerwise Offload](feature/offloader/distributed_layerwise_offload.md) streams diffusion blocks from host memory within the existing parallel topology. diff --git a/docs/design/feature/cpu_offloading.md b/docs/design/feature/offloader/README.md similarity index 86% rename from docs/design/feature/cpu_offloading.md rename to docs/design/feature/offloader/README.md index c37b705d2..76bdad14a 100644 --- a/docs/design/feature/cpu_offloading.md +++ b/docs/design/feature/offloader/README.md @@ -5,9 +5,9 @@ The three strategies have separate user and design pages: | Strategy | User guide | Design contract | | --- | --- | --- | -| Model-level | [Guide](../../user_guide/diffusion/offloader/cpu_offload.md) | [Design](model_level_offload.md) | -| Layerwise | [Guide](../../user_guide/diffusion/offloader/layerwise_offload.md) | [Design](layerwise_offload.md) | -| Distributed layerwise | [Guide](../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) | [Design](distributed_layerwise_offload.md) | +| Model-level | [Guide](../../../user_guide/diffusion/offloader/cpu_offload.md) | [Design](cpu_offload.md) | +| Layerwise | [Guide](../../../user_guide/diffusion/offloader/layerwise_offload.md) | [Design](layerwise_offload.md) | +| Distributed layerwise | [Guide](../../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) | [Design](distributed_layerwise_offload.md) | ## Strategy selection @@ -76,5 +76,5 @@ ownership. - Platform streams, events, synchronization, and cache management go through the vLLM-Omni platform abstraction. -The diffusion [Offloader module design](../module/diffusion/offloader.md) +The diffusion [Offloader module design](../../module/diffusion/offloader.md) describes how these feature contracts fit into the larger diffusion runtime. diff --git a/docs/design/feature/model_level_offload.md b/docs/design/feature/offloader/cpu_offload.md similarity index 94% rename from docs/design/feature/model_level_offload.md rename to docs/design/feature/offloader/cpu_offload.md index b7e80d04d..2318611cc 100644 --- a/docs/design/feature/model_level_offload.md +++ b/docs/design/feature/offloader/cpu_offload.md @@ -2,7 +2,7 @@ Model-level offload enforces mutual exclusion between pipeline component groups. For commands and model support, see the -[model-level user guide](../../user_guide/diffusion/offloader/cpu_offload.md). +[model-level user guide](../../../user_guide/diffusion/offloader/cpu_offload.md). ## Generic hook path @@ -53,4 +53,4 @@ attribute-name heuristics. residency is restored. Shared selection and lifecycle behavior is defined in the -[CPU Offloading design](cpu_offloading.md). +[CPU Offloading design](README.md). diff --git a/docs/design/feature/distributed_layerwise_offload.md b/docs/design/feature/offloader/distributed_layerwise_offload.md similarity index 97% rename from docs/design/feature/distributed_layerwise_offload.md rename to docs/design/feature/offloader/distributed_layerwise_offload.md index 0d260ca25..ade39fa27 100644 --- a/docs/design/feature/distributed_layerwise_offload.md +++ b/docs/design/feature/offloader/distributed_layerwise_offload.md @@ -7,7 +7,7 @@ either shard those host-side weights across an existing parallel group or keep the standard loader's rank-local weights and avoid an additional collective. For user-facing commands, see the -[distributed layerwise offloading guide](../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) +[distributed layerwise offloading guide](../../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md). ## Status @@ -56,7 +56,7 @@ AllGather: [full N+1] [full N+2] Buffers: [current slot] [prefetch slot] [current slot] ``` -![DLO double-buffer prefetch pipeline](../figures/dlo/dlo_pipeline.gif) +![DLO double-buffer prefetch pipeline](../../figures/dlo/dlo_pipeline.gif) The backend uses two shared device buffers, so accelerator weight residency is bounded by the largest streamed blocks rather than the complete model. diff --git a/docs/design/feature/layerwise_offload.md b/docs/design/feature/offloader/layerwise_offload.md similarity index 94% rename from docs/design/feature/layerwise_offload.md rename to docs/design/feature/offloader/layerwise_offload.md index 94e9ff1b9..3fff501da 100644 --- a/docs/design/feature/layerwise_offload.md +++ b/docs/design/feature/offloader/layerwise_offload.md @@ -2,7 +2,7 @@ Layerwise offload streams transformer blocks through one accelerator. For commands and model support, see the -[layerwise user guide](../../user_guide/diffusion/offloader/layerwise_offload.md). +[layerwise user guide](../../../user_guide/diffusion/offloader/layerwise_offload.md). ## Hook ring @@ -54,4 +54,4 @@ streamed by this backend. [Distributed Layerwise Offload](distributed_layerwise_offload.md). Shared selection and lifecycle behavior is defined in the -[CPU Offloading design](cpu_offloading.md). +[CPU Offloading design](README.md). diff --git a/docs/design/feature/quantization.md b/docs/design/feature/quantization.md index 37dd5e26b..7ad612d12 100644 --- a/docs/design/feature/quantization.md +++ b/docs/design/feature/quantization.md @@ -186,7 +186,7 @@ Distributed layerwise offload has an additional boundary: its sharded mmap AllGather path is not compatible with online quantization because the path expects already materialized checkpoint weights. Use the rank-local `--dlo-no-use-allgather` path or a compatible pre-quantized checkpoint when -combining these features. See the [DLO design document](distributed_layerwise_offload.md) +combining these features. See the [DLO design document](offloader/distributed_layerwise_offload.md) for the full compatibility matrix. ## Adding a quantization backend diff --git a/docs/design/index.md b/docs/design/index.md index 798201214..0db152335 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -61,10 +61,10 @@ The design contracts separate selection mechanics from backend algorithms: #### CPU offloading -- [Overview and Shared Contracts](feature/cpu_offloading.md) -- [Model-Level Offload](feature/model_level_offload.md) -- [Layerwise Offload](feature/layerwise_offload.md) -- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md) +- [Overview and Shared Contracts](feature/offloader/README.md) +- [Model-Level Offload](feature/offloader/cpu_offload.md) +- [Layerwise Offload](feature/offloader/layerwise_offload.md) +- [Distributed Layerwise Offload](feature/offloader/distributed_layerwise_offload.md) - [Cache-DiT](feature/cache_dit.md) - [TeaCache](feature/teacache.md) diff --git a/docs/features/README.md b/docs/features/README.md index 411a31164..8090b783f 100644 --- a/docs/features/README.md +++ b/docs/features/README.md @@ -48,7 +48,7 @@ defines the shared configuration and backend extension points. | Goal | User guide | Related design contract | | --- | --- | --- | | Compare acceleration methods and supported combinations | [Overview](../user_guide/diffusion_features.md), [Feature Compatibility](../user_guide/feature_compatibility.md) | [Diffusion acceleration designs](../design/index.md#diffusion-acceleration) | -| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [CPU Offloading](../design/feature/cpu_offloading.md) | +| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [CPU Offloading](../design/feature/offloader/README.md) | | Reuse denoising computation | [Cache Acceleration](../user_guide/diffusion/cache_acceleration/cache_dit.md) | [Cache-DiT](../design/feature/cache_dit.md), [TeaCache](../design/feature/teacache.md) | | Distribute diffusion work across devices | [Parallelism](../user_guide/diffusion/parallelism/overview.md) | [Parallelism designs](../design/index.md#parallelism) | | Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Attention Backend Selection](../design/feature/attention_backend_selection.md) | diff --git a/docs/user_guide/diffusion/cpu_offload.md b/docs/user_guide/diffusion/cpu_offload.md index c1396f12b..a17e4a114 100644 --- a/docs/user_guide/diffusion/cpu_offload.md +++ b/docs/user_guide/diffusion/cpu_offload.md @@ -5,7 +5,7 @@ the coarsest strategy that meets your memory target: finer-grained streaming saves more device memory but adds more transfer and setup overhead. For the shared factory, discovery, and lifecycle contract, see the -[CPU Offloading design](../../design/feature/cpu_offloading.md). +[CPU Offloading design](../../design/feature/offloader/README.md). ## Choose a strategy diff --git a/docs/user_guide/diffusion/offloader/cpu_offload.md b/docs/user_guide/diffusion/offloader/cpu_offload.md index 017358e89..558985e66 100644 --- a/docs/user_guide/diffusion/offloader/cpu_offload.md +++ b/docs/user_guide/diffusion/offloader/cpu_offload.md @@ -77,5 +77,5 @@ while reusing sequential `.to()` movers. - Higher cold-start latency. - Transfers between encoder and denoising phases add latency. -See the [model-level design](../../../design/feature/model_level_offload.md) +See the [model-level design](../../../design/feature/offloader/cpu_offload.md) for lifecycle and extension invariants. diff --git a/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md index 9f5ae3493..15298c897 100644 --- a/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md +++ b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md @@ -5,7 +5,7 @@ deployments. With AllGather enabled, each rank stores roughly `1 / dp_size` of the host weights and reconstructs each layer at runtime. Without AllGather, each rank streams its standard-loader rank-local weights independently. -See the [DLO feature design](../../../design/feature/distributed_layerwise_offload.md) +See the [DLO feature design](../../../design/feature/offloader/distributed_layerwise_offload.md) for the implementation contract and compatibility matrix. ## Execution model diff --git a/docs/user_guide/diffusion/offloader/layerwise_offload.md b/docs/user_guide/diffusion/offloader/layerwise_offload.md index 041278b80..c88b67fcc 100644 --- a/docs/user_guide/diffusion/offloader/layerwise_offload.md +++ b/docs/user_guide/diffusion/offloader/layerwise_offload.md @@ -53,7 +53,7 @@ class Flux2Transformer2DModel(nn.Module): ] ``` -See the [layerwise design](../../../design/feature/layerwise_offload.md) for +See the [layerwise design](../../../design/feature/offloader/layerwise_offload.md) for the discovery and hook invariants. `OffloadPlan` is a separate declarative topology path for distributed layerwise offload. diff --git a/mkdocs.yml b/mkdocs.yml index fd97b9b37..0b65aadd4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -79,6 +79,10 @@ plugins: user_guide/quantization/index.md: user_guide/quantization/overview.md design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md + design/feature/cpu_offloading.md: design/feature/offloader/README.md + design/feature/model_level_offload.md: design/feature/offloader/cpu_offload.md + design/feature/layerwise_offload.md: design/feature/offloader/layerwise_offload.md + design/feature/distributed_layerwise_offload.md: design/feature/offloader/distributed_layerwise_offload.md - glightbox - git-revision-date-localized: # exclude files