[Feat] support hsdp for Bagel (#3150)
Signed-off-by: siyuan.lei <siyuanlei37@gmail.com> Signed-off-by: lsyyyyy <siyuanlei37@gmail.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com> Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
This commit is contained in:
1 parent
ea4cf77f56
commit
32663f21d5
7 files changed
+131
-32
No files matched your search
@@ -195,6 +195,25 @@ stages:
|
||||
devices: "0,1"
|
||||
```
|
||||
|
||||
#### Hybrid Sharded Data Parallel (HSDP)
|
||||
|
||||
For larger Bagel deployments on multiple GPUs, you can enable HSDP (Hybrid Sharded Data Parallel) by modifying the stage configuration (for example, [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml)). HSDP shards transformer weights across GPUs to reduce per-GPU memory usage.
|
||||
|
||||
1. **Enable HSDP**: Set `use_hsdp: true`.
|
||||
2. **Set shard size**: Set `hsdp_shard_size` to the number of GPUs used for sharding (for example, `4`).
|
||||
3. **Set replicate size**: Usually keep `hsdp_replicate_size: 1` unless you want replicated HSDP groups.
|
||||
4. **Set devices**: Specify the comma-separated GPU IDs used by the diffusion stage (for example, `"0,1,2,3"`).
|
||||
|
||||
Example configuration for HSDP across 4 GPUs:
|
||||
```yaml
|
||||
- stage_id: 1
|
||||
devices: "0,1,2,3"
|
||||
parallel_config:
|
||||
use_hsdp: true
|
||||
hsdp_shard_size: 4
|
||||
hsdp_replicate_size: 1
|
||||
```
|
||||
|
||||
Then pass the custom deploy YAML:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -68,6 +68,25 @@ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --tensor-parallel-size
|
||||
|
||||
Or set `tensor_parallel_size` per stage in a custom deploy YAML.
|
||||
|
||||
#### Hybrid Sharded Data Parallel (HSDP)
|
||||
|
||||
For larger Bagel deployments on multiple GPUs, you can enable HSDP (Hybrid Sharded Data Parallel) by modifying the stage configuration (for example, [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml)). HSDP shards transformer weights across GPUs to reduce per-GPU memory usage.
|
||||
|
||||
1. **Enable HSDP**: Set `use_hsdp: true`.
|
||||
2. **Set shard size**: Set `hsdp_shard_size` to the number of GPUs used for sharding (for example, `4`).
|
||||
3. **Set replicate size**: Usually keep `hsdp_replicate_size: 1` unless you want replicated HSDP groups.
|
||||
4. **Set devices**: Specify the comma-separated GPU IDs used by the diffusion stage (for example, `"0,1,2,3"`).
|
||||
|
||||
Example configuration for HSDP across 4 GPUs:
|
||||
```yaml
|
||||
- stage_id: 1
|
||||
devices: "0,1,2,3"
|
||||
parallel_config:
|
||||
use_hsdp: true
|
||||
hsdp_shard_size: 4
|
||||
hsdp_replicate_size: 1
|
||||
```
|
||||
|
||||
### Multi-Node Deployment
|
||||
|
||||
Deploy each stage on a **separate node** for better resource utilization. Replace `<ORCHESTRATOR_IP>` with the actual IP address of your orchestrator node.
|
||||
|
||||
Reference in new issue
Block a user