[REFACT]Refactor diffusion parallel state (#5531)
Signed-off-by: bjf-frz <frz123db@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
14f2c16fb2
commit
14bd8f8772
19 files changed
+315
-387
No files matched your search
@@ -152,7 +152,8 @@ In `DiffusionParallelConfig`:
|
||||
**Constraints:**
|
||||
|
||||
- `hsdp_replicate_size × hsdp_shard_size == world_size`
|
||||
- HSDP cannot be used with Tensor Parallelism (`tensor_parallel_size` must be 1)
|
||||
- HSDP cannot be used with Tensor Parallelism, ordinary Data Parallelism,
|
||||
Pipeline Parallelism, or Expert Parallelism.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user