[skip ci][Doc] Refine the Diffusion Features User Guide (#1928)
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
This commit is contained in:
Didan Deng authored and GitHub committed
1 parent
c9370f493e
commit
911be0d426
20 files changed
+1912
-1244
No files matched your search
@@ -0,0 +1,169 @@
|
||||
# CFG-Parallel Guide
|
||||
|
||||
|
||||
## Table of Content
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Example Script](#example-script)
|
||||
- [Configuration Parameters](#configuration-parameters)
|
||||
- [Best Practices](#best-practices)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Summary](#summary)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
CFG-Parallel accelerates diffusion models by distributing positive and negative classifier-free guidance (CFG) passes across different GPUs, providing ~1.8x speedup when CFG is enabled. It's ideal for image editing tasks that require guidance scales greater than 1.0.
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simplest working example:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from PIL import Image
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image-Edit",
|
||||
parallel_config=DiffusionParallelConfig(cfg_parallel_size=2), # Enable CFG-Parallel
|
||||
)
|
||||
|
||||
input_image = Image.open("input.png").convert("RGB")
|
||||
outputs = omni.generate(
|
||||
{
|
||||
"prompt": "turn this cat to a dog",
|
||||
"negative_prompt": "low quality, blurry",
|
||||
"multi_modal_data": {"image": input_image},
|
||||
},
|
||||
OmniDiffusionSamplingParams(
|
||||
true_cfg_scale=4.0,
|
||||
num_inference_steps=50,
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Script
|
||||
|
||||
### Offline Inference
|
||||
|
||||
Use python script under `examples/offline_inference/image_to_image/image_edit.py`:
|
||||
|
||||
```bash
|
||||
cd examples/offline_inference/image_to_image/
|
||||
python image_edit.py \
|
||||
--model "Qwen/Qwen-Image-Edit" \
|
||||
--image "input.png" \
|
||||
--prompt "turn this cat to a dog" \
|
||||
--negative-prompt "low quality, blurry" \
|
||||
--cfg-scale 4.0 \
|
||||
--output "edited_image.png" \
|
||||
--cfg-parallel-size 2
|
||||
```
|
||||
|
||||
### Online Serving
|
||||
|
||||
Enable CFG-Parallel in online serving:
|
||||
|
||||
```bash
|
||||
# Default configuration
|
||||
vllm serve Qwen/Qwen-Image-Edit --omni --port 8091 --cfg-parallel-size 2
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
In `DiffusionParallelConfig`
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `cfg_parallel_size` | int | 1 | Number of GPUs for CFG parallelism. Set to 2 to enable CFG-Parallel (rank 0 for positive, rank 1 for negative branch) |
|
||||
|
||||
|
||||
!!! info
|
||||
Most models support `cfg_parallel_size=2` (positive branch on rank 0, negative branch on rank 1). **Bagel** is an exception: it supports `cfg_parallel_size=3`, which adds a third branch on rank 2 for full three-way CFG parallelism.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use
|
||||
|
||||
**Good for:**
|
||||
|
||||
- Tasks requiring classifier-free guidance
|
||||
- Multi-GPU setups (at least 2 GPUs available)
|
||||
- Combining with other parallelism methods (sequence/tensor parallel)
|
||||
|
||||
**Not for:**
|
||||
|
||||
- Single GPU setups
|
||||
- Models that don't support CFG-Parallel (check [supported models](../../diffusion_features.md#supported-models))
|
||||
- Workloads without negative prompts or classifier-free guidance
|
||||
- Very short inference runs (< 10 steps) where parallelism overhead may outweigh benefits
|
||||
|
||||
### Expected Performance
|
||||
|
||||
| Configuration | Speedup | Quality | Use Case |
|
||||
|--------------|---------|---------|----------|
|
||||
| CFG-Parallel (2 GPUs) | 1.5~1.8x | No degradation | Large model, VRAM limited |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issue 1: No Speedup with CFG-Parallel
|
||||
|
||||
**Symptoms**: CFG-Parallel enabled but no performance improvement
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Ensure CFG scale is set correctly:**
|
||||
```python
|
||||
# Bad: No CFG effect
|
||||
sampling_params = OmniDiffusionSamplingParams(num_inference_steps=50)
|
||||
|
||||
# Good: CFG-Parallel will work
|
||||
sampling_params = OmniDiffusionSamplingParams(
|
||||
num_inference_steps=50,
|
||||
true_cfg_scale=4.0 # Must be > 1.0
|
||||
)
|
||||
```
|
||||
|
||||
2. **Add negative prompt:**
|
||||
```python
|
||||
outputs = omni.generate(
|
||||
{
|
||||
"prompt": "beautiful landscape",
|
||||
"negative_prompt": "low quality, blurry", # Required for best results
|
||||
"multi_modal_data": {"image": input_image}
|
||||
},
|
||||
sampling_params
|
||||
)
|
||||
```
|
||||
|
||||
3. **Check model support:**
|
||||
- Verify your model in [supported models](../../diffusion_features.md#supported-models)
|
||||
- Some models don't support CFG-Parallel
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
1. ✅ **Enable CFG-Parallel** - Set `cfg_parallel_size=2` in `DiffusionParallelConfig` to get speedup when using CFG
|
||||
2. ✅ **Set CFG Scale** - Ensure `true_cfg_scale > 1.0` in `OmniDiffusionSamplingParams` for CFG-Parallel to take effect
|
||||
3. ✅ **Check Model Support** - Verify your model supports CFG-Parallel in [supported models](../../diffusion_features.md#supported-models)
|
||||
@@ -0,0 +1,87 @@
|
||||
# Expert Parallelism Guide
|
||||
|
||||
|
||||
## Table of Content
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Configuration Parameters](#configuration-parameters)
|
||||
- [Best Practices](#best-practices)
|
||||
- [Summary](#summary)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Unlike Tensor Parallelism which shards every layer's weights, Expert Parallelism (EP) only shards the MoE expert MLP blocks. This significantly reduces the memory footprint of MoE models (e.g., HunyuanImage3.0) while maintaining constant dense-equivalent compute efficiency.
|
||||
|
||||
During the forward pass, a gating mechanism routes tokens to their designated experts, requiring all-to-all communication to dispatch tokens to the correct ranks and combine results.
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
!!! note "EP Size Constraint"
|
||||
The effective EP size equals `tp × sp × cfg × dp`. At least one of TP/SP/CFG/DP must be set when EP is enabled.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
omni = Omni(
|
||||
model="tencent/HunyuanImage-3.0",
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
tensor_parallel_size=8,
|
||||
enable_expert_parallel=True,
|
||||
),
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
"A brown and white dog is running on the grass",
|
||||
OmniDiffusionSamplingParams(
|
||||
num_inference_steps=50,
|
||||
width=1024,
|
||||
height=1024,
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
In `DiffusionParallelConfig`:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `enable_expert_parallel` | bool | False | Enable Expert Parallelism for MoE models |
|
||||
|
||||
EP size is derived automatically as `tp × sp × cfg × dp` — configure at least one of those to set the EP degree.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use
|
||||
|
||||
**Good for:**
|
||||
|
||||
- MoE models (e.g., HunyuanImage3.0) with numbers of experts
|
||||
- Memory-constrained multi-GPU setups where only expert blocks need sharding
|
||||
|
||||
**Not for:**
|
||||
|
||||
- Dense models (no MoE layers) — EP has no effect
|
||||
- Single GPU setups
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
1. ✅ **Enable EP** - Set `enable_expert_parallel=True` in `DiffusionParallelConfig` for MoE models
|
||||
2. ✅ **Set parallelism degree** - At least one of `tensor_parallel_size` / `ulysses_degree` / `cfg_parallel_size` must be > 1 to define the EP size
|
||||
@@ -0,0 +1,149 @@
|
||||
# HSDP Guide
|
||||
|
||||
|
||||
## Table of Content
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Example Script](#example-script)
|
||||
- [Configuration Parameters](#configuration-parameters)
|
||||
- [Best Practices](#best-practices)
|
||||
- [Summary](#summary)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
HSDP (Hybrid Sharded Data Parallel) shards model weights across GPUs to reduce per-GPU memory usage. This enables inference of large models (e.g., Wan2.2 14B) on GPUs with limited memory.
|
||||
|
||||
Unlike Tensor Parallelism which splits computation, HSDP uses PyTorch's FSDP2 to shard and redistribute weights at runtime. Each GPU only holds a fraction of the model weights, and weights are gathered on-demand during forward passes.
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
**Operating Modes:**
|
||||
|
||||
- **Standalone Mode**: HSDP alone without other parallelism. Must specify `hsdp_shard_size` explicitly.
|
||||
- **Combined Mode**: HSDP overlays on top of other parallelism (Ulysses-SP, CFG-Parallel). HSDP dimensions must match world_size.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simplest working example (standalone HSDP, shard across 4 GPUs):
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
use_hsdp=True,
|
||||
hsdp_shard_size=4, # Shard across 4 GPUs
|
||||
),
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
"A cat playing piano",
|
||||
OmniDiffusionSamplingParams(num_inference_steps=50),
|
||||
)
|
||||
```
|
||||
|
||||
### Combined with Sequence Parallel
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
ulysses_degree=4, # Sequence parallel
|
||||
use_hsdp=True, # HSDP overlays on SP
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Script
|
||||
|
||||
### Offline Inference
|
||||
|
||||
Use Python script under `examples/offline_inference/image_to_video/`:
|
||||
|
||||
```bash
|
||||
# Standalone HSDP: shard across 4 GPUs
|
||||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--use-hsdp \
|
||||
--hsdp-shard-size 4
|
||||
|
||||
# Combined HSDP + Sequence Parallel
|
||||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--ulysses-degree 4 \
|
||||
--use-hsdp
|
||||
```
|
||||
|
||||
### Online Serving
|
||||
|
||||
**Standalone HSDP** (shard model across 4 GPUs):
|
||||
|
||||
```bash
|
||||
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091 \
|
||||
--use-hsdp --hsdp-shard-size 4
|
||||
```
|
||||
|
||||
**Combined with Sequence Parallel**:
|
||||
|
||||
```bash
|
||||
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091 \
|
||||
--use-hsdp --usp 4
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
In `DiffusionParallelConfig`:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `use_hsdp` | bool | False | Enable HSDP |
|
||||
| `hsdp_shard_size` | int | -1 | Number of GPUs to shard weights across. `-1` = auto (requires other parallelism > 1) |
|
||||
| `hsdp_replicate_size` | int | 1 | Number of replica groups. Each group holds a full sharded copy |
|
||||
|
||||
**Constraints:**
|
||||
|
||||
- `hsdp_replicate_size × hsdp_shard_size == world_size`
|
||||
- HSDP cannot be used with Tensor Parallelism (`tensor_parallel_size` must be 1)
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use
|
||||
|
||||
**Good for:**
|
||||
|
||||
- Very large models (e.g., Wan2.2 14B)
|
||||
- Multi-GPU setups where memory reduction is the primary goal
|
||||
- Combining with Sequence Parallelism for large video models
|
||||
|
||||
**Not for:**
|
||||
|
||||
- Models that fit comfortably in single-GPU memory
|
||||
- Use cases requiring Tensor Parallelism (HSDP and TP are mutually exclusive)
|
||||
|
||||
### Adding HSDP Support to New Models
|
||||
|
||||
For detailed instructions on adding HSDP support to new models, see the [HSDP Contributing Guide](../../../design/feature/hsdp.md).
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
1. ✅ **Enable HSDP** - Set `use_hsdp=True` and `hsdp_shard_size` to reduce per-GPU memory for large models
|
||||
2. ✅ **Combine with SP** - Use together with `ulysses_degree` for video models requiring both memory reduction and sequence parallelism
|
||||
3. ⚠️ **Incompatible with TP** - `tensor_parallel_size` must be 1 when HSDP is enabled
|
||||
@@ -0,0 +1,16 @@
|
||||
# Parallelism Acceleration Guide
|
||||
|
||||
This guide covers the parallelism methods in vLLM-Omni for speeding up diffusion model inference and reducing per-device memory requirements.
|
||||
|
||||
## Supported Methods
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| **[Tensor Parallelism](tensor_parallel.md)** | Shards DiT weights across GPUs to reduce per-GPU memory |
|
||||
| **[Sequence Parallelism](sequence_parallel.md)** | Splits sequence dimension across GPUs (Ulysses-SP, Ring-Attention, or hybrid) for high-resolution images and videos |
|
||||
| **[CFG-Parallel](cfg_parallel.md)** | Runs CFG positive/negative branches on separate GPUs for ~1.8x speedup on guided generation |
|
||||
| **[VAE Patch Parallelism](vae_patch_parallel.md)** | Distributes VAE decode spatially across GPUs to reduce peak VAE memory |
|
||||
| **[HSDP](hsdp.md)** | Shards full model weights via PyTorch FSDP2 to enable large-model inference on memory-constrained GPUs |
|
||||
| **[Expert Parallelism](expert_parallel.md)** | Shards MoE expert blocks across GPUs for MoE models (e.g. HunyuanImage3.0) |
|
||||
|
||||
See [Supported Models](../../diffusion_features.md#supported-models) for per-model compatibility.
|
||||
@@ -0,0 +1,233 @@
|
||||
# Sequence Parallelism Guide
|
||||
|
||||
|
||||
## Table of Content
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Example Script](#example-script)
|
||||
- [Best Practices](#best-practices)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Summary](#summary)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Sequence parallelism splits the input along the sequence dimension across multiple GPUs, allowing each device to process only a portion of the sequence. vLLM-Omni provides 1.5x-3.6x speedup for large images and videos using DeepSpeed Ulysses, Ring-Attention, or hybrid approaches. Use sequence parallelism when generating high-resolution images/videos that don't fit on a single GPU or require faster inference.
|
||||
|
||||
See supported models list in [Diffusion Features - Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
**Supported Methods:**
|
||||
|
||||
- **DeepSpeed Ulysses Sequence Parallel (Ulysses-SP)** ([paper](https://arxiv.org/pdf/2309.14509)): Uses all-to-all communication for subset of attention heads per device
|
||||
- **Ring-Attention** ([paper](https://arxiv.org/abs/2310.01889)): Uses ring-based P2P communication with sharded sequence dimension throughout
|
||||
- **Hybrid Ulysses + Ring**: Combines both for larger scale parallelism (`ulysses_degree × ring_degree`)
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage - Ulysses-SP
|
||||
|
||||
Simplest working example with Ulysses Sequence Parallel:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
parallel_config=DiffusionParallelConfig(ulysses_degree=2) # Enable Ulysses-SP
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
"A cat sitting on a windowsill",
|
||||
OmniDiffusionSamplingParams(num_inference_steps=50, width=1024, height=1024),
|
||||
)
|
||||
```
|
||||
|
||||
!!! note "Experimental UAA mode"
|
||||
`ulysses_mode="advanced_uaa"` is an experimental extension to Ulysses-SP. It lets Ulysses attention handle arbitrary sequence lengths and arbitrary attention head counts without relying on `attention_mask`-based token padding.
|
||||
|
||||
In hybrid Ulysses + Ring mode, Ring still requires every rank in the same ring group to observe the same post-Ulysses sequence length. If that condition is not met, vLLM-Omni raises a validation error instead of entering the ring kernel with inconsistent shapes.
|
||||
|
||||
To enable the experimental UAA mode, use a model/configuration that requires it. For example, `Tongyi-MAI/Z-Image-Turbo` has 30 attention heads, so `ulysses_degree=4` requires UAA because 30 is not divisible by 4:
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Tongyi-MAI/Z-Image-Turbo",
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
ulysses_degree=4,
|
||||
ulysses_mode="advanced_uaa",
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
### Alternative Methods
|
||||
|
||||
**Ring-Attention** (better for very long sequences):
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
parallel_config=DiffusionParallelConfig(ring_degree=2) # Enable Ring-Attention
|
||||
)
|
||||
```
|
||||
|
||||
**Hybrid Ulysses + Ring** (for larger scale):
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
parallel_config=DiffusionParallelConfig(ulysses_degree=2, ring_degree=2) # 4 GPUs total
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Script
|
||||
|
||||
### Offline Inference
|
||||
|
||||
Use Python script under `examples/offline_inference/text_to_image/text_to_image.py`:
|
||||
|
||||
**Ulysses-SP:**
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--prompt "A cat sitting on a windowsill" \
|
||||
--ulysses-degree 2 \
|
||||
--width 1024 --height 1024
|
||||
```
|
||||
|
||||
**Ring-Attention:**
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--prompt "A cat sitting on a windowsill" \
|
||||
--ring-degree 2 \
|
||||
--width 1024 --height 1024
|
||||
```
|
||||
|
||||
**Hybrid Ulysses + Ring:**
|
||||
|
||||
```bash
|
||||
# Hybrid: 2 Ulysses × 2 Ring = 4 GPUs total
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--prompt "A cat sitting on a windowsill" \
|
||||
--ulysses-degree 2 --ring-degree 2 \
|
||||
--width 1024 --height 1024
|
||||
```
|
||||
|
||||
### Online Serving
|
||||
|
||||
**Ulysses-SP:**
|
||||
|
||||
```bash
|
||||
# Text-to-image (requires >= 2 GPUs)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2
|
||||
```
|
||||
|
||||
**Ulysses-SP with UAA mode** (for models with non-divisible head counts):
|
||||
|
||||
```bash
|
||||
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091 --usp 4 --ulysses-mode advanced_uaa
|
||||
```
|
||||
|
||||
**Ring-Attention:**
|
||||
|
||||
```bash
|
||||
# Text-to-image (requires >= 2 GPUs)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 --ring 2
|
||||
```
|
||||
|
||||
**Hybrid Ulysses + Ring:**
|
||||
|
||||
```bash
|
||||
# Text-to-image (requires >= 4 GPUs)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
In `DiffusionParallelConfig`:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `ulysses_degree` | int | 1 | Number of GPUs for Ulysses-SP. Uses all-to-all communication. |
|
||||
| `ring_degree` | int | 1 | Number of GPUs for Ring-Attention. Uses P2P ring communication. |
|
||||
| `ulysses_mode` | str | `"default"` | Ulysses attention mode. Set to `"advanced_uaa"` to handle arbitrary sequence lengths and head counts without padding. |
|
||||
|
||||
**Notes:**
|
||||
- Total sequence parallel size equals to `ulysses_degree × ring_degree`
|
||||
- Degrees must evenly divide the sequence length for optimal performance (or use `ulysses_mode="advanced_uaa"` for Ulysses-SP)
|
||||
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use
|
||||
|
||||
**Good for:**
|
||||
|
||||
- Large images (1024x1024 or higher) or videos
|
||||
- Fast inter-GPU communication, larger bandwidth (e.g., NVLink)
|
||||
|
||||
**Not for:**
|
||||
|
||||
- Small images (<1024px) - overhead exceeds benefit, use single GPU with cache instead
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issue 1: Performance Not Scaling
|
||||
|
||||
**Symptoms**: Adding GPUs doesn't improve speed proportionally, or higher parallelism degree is slower
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check GPU topology
|
||||
nvidia-smi topo -m
|
||||
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Check inter-GPU communication - NVLink is better than PCIe
|
||||
2. Reduce parallelism degree if over-parallelized:
|
||||
```python
|
||||
# If 4 GPUs is slower than 2
|
||||
parallel_config=DiffusionParallelConfig(ulysses_degree=2)
|
||||
```
|
||||
3. Try to switch between Ring-Attention and Ulysses-SP
|
||||
|
||||
- Ring-Attention has advantages, like communication-computation overlap, but the block-wise loop overhead is relatively higher, especially for short sequences
|
||||
- Ulysses-SP: can benefit from larger bandwidth (such as NVLink), with two major constraints, the sequence length should be divisible by usp size, and the number of heads should be divisible by usp size (or use `ulysses_mode="advanced_uaa"`)
|
||||
|
||||
|
||||
### Common Issue 2: Out of Memory (OOM)
|
||||
|
||||
**Symptoms**: CUDA OOM errors or process crashes with memory errors
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Increase parallelism degree to split sequence more:
|
||||
```python
|
||||
parallel_config=DiffusionParallelConfig(ulysses_degree=4) # From 2
|
||||
```
|
||||
2. Combine with other parallelism method, e.g., tensor parallel, and memory optimization methods, e.g., cpu offloading.
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
1. ✅ **Enable Sequence Parallelism** - Set `ulysses_degree` or `ring_degree` for long sequence generation
|
||||
2. ✅ **UAA mode** - Use `ulysses_mode="advanced_uaa"` when head count is not divisible by `ulysses_degree`
|
||||
3. ✅ **Troubleshooting** - Check GPU topology with `nvidia-smi topo -m`, reduce degree if performance doesn't scale
|
||||
@@ -0,0 +1,151 @@
|
||||
# Tensor Parallelism Guide
|
||||
|
||||
|
||||
## Table of Content
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Example Script](#example-script)
|
||||
- [Configuration Parameters](#configuration-parameters)
|
||||
- [Best Practices](#best-practices)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Summary](#summary)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Tensor Parallelism (TP) shards some model weights across multiple GPUs, usually the Linear layers. This enables running large models that don't fit on a single GPU. It's essential for memory-constrained setups or very large models.
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
!!! note "TP Limitations for Diffusion Models"
|
||||
We currently implement Tensor Parallelism (TP) only for the DiT (Diffusion Transformer) blocks. This is because the `text_encoder` component in vLLM-Omni uses the original Transformers implementation, which does not yet support TP.
|
||||
|
||||
- Good news: The text_encoder typically has minimal impact on overall inference performance.
|
||||
- Bad news: When TP is enabled, every TP process retains a full copy of the text_encoder weights, leading to significant GPU memory waste.
|
||||
|
||||
We are actively refactoring this design to address this. For details and progress, please refer to [Issue #771](https://github.com/vllm-project/vllm-omni/issues/771).
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simplest working example:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
omni = Omni(
|
||||
model="Tongyi-MAI/Z-Image-Turbo",
|
||||
parallel_config=DiffusionParallelConfig(tensor_parallel_size=2), # Enable TP
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
"a cat reading a book",
|
||||
OmniDiffusionSamplingParams(num_inference_steps=9),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Script
|
||||
|
||||
### Offline Inference
|
||||
|
||||
Use Python script under `examples/offline_inference`, and enable TP:
|
||||
|
||||
```bash
|
||||
# Text-to-Image with Qwen-Image
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--tensor-parallel-size 2
|
||||
|
||||
# Image Editing with Qwen-Image-Edit
|
||||
python examples/offline_inference/image_to_image/image_edit.py \
|
||||
--model Qwen/Qwen-Image-Edit \
|
||||
--image input.png \
|
||||
--prompt "Edit description" \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
### Online Serving
|
||||
|
||||
You can enable tensor parallelism in online serving via `--tensor-parallel-size`:
|
||||
|
||||
```bash
|
||||
# Text-to-Image with Qwen-Image on 2 GPUs
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 \
|
||||
--tensor-parallel-size 2
|
||||
|
||||
# Text-to-Image with Z-Image (TP=2 only)
|
||||
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091 \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
In `DiffusionParallelConfig`:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `tensor_parallel_size` | int | 1 | Number of GPUs to shard model weights across. Must divide number of heads. |
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use
|
||||
|
||||
**Good for:**
|
||||
|
||||
- Large models that don't fit on a single GPU, especially for models with large DiT blocks (transformer layers)
|
||||
- Memory-constrained environments
|
||||
|
||||
**Not for:**
|
||||
|
||||
- When maximum throughput is needed and memory is sufficient
|
||||
- Models with incompatible dimensions (e.g., Z-Image `num_heads=30`, which now supports `tensor_parallel_size=2`)
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issue 1: Out of Memory (OOM)
|
||||
|
||||
**Symptoms**: CUDA OOM errors during model loading or inference, process crashes with memory errors
|
||||
|
||||
**Solution**:
|
||||
```python
|
||||
# Step 1: Enable TP with smallest degree
|
||||
parallel_config=DiffusionParallelConfig(tensor_parallel_size=2)
|
||||
|
||||
# Step 2: If still OOM, increase TP degree
|
||||
parallel_config=DiffusionParallelConfig(tensor_parallel_size=4)
|
||||
|
||||
```
|
||||
|
||||
### Common Issue 2: Divisibility Error
|
||||
|
||||
**Symptoms**: Error like "Model dimension X not divisible by tensor_parallel_size Y"
|
||||
|
||||
**Solutions**:
|
||||
1. Check model-specific constraints (e.g., Z-Image only supports TP=2)
|
||||
2. Use a smaller TP size that divides model dimensions
|
||||
3. Consult [Supported Models](../../diffusion_features.md#supported-models) for compatible TP sizes
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
1. ✅ **Enable TP** - Set `--tensor-parallel-size` to reduce per-GPU memory
|
||||
2. ✅ **Increase TP size** - Only increase if OOM persists
|
||||
3. ⚠️ **Text encoder not sharded** - Known limitation
|
||||
@@ -0,0 +1,200 @@
|
||||
# VAE Patch Parallelism Guide
|
||||
|
||||
|
||||
## Table of Content
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Example Script](#example-script)
|
||||
- [Configuration Parameters](#configuration-parameters)
|
||||
- [Best Practices](#best-practices)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Summary](#summary)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
VAE Patch Parallelism distributes the VAE (Variational AutoEncoder) decode/encode computation across multiple GPUs by splitting the latent space into spatial tiles or patches. Each GPU processes a subset of tiles in parallel, significantly reducing peak memory consumption during the VAE decode stage while maintaining output quality.
|
||||
|
||||
This is particularly useful for:
|
||||
- **High-resolution image generation** where VAE decode can become a memory bottleneck
|
||||
- **Memory-constrained environments** where the VAE decode activation peak exceeds available VRAM
|
||||
- **Multi-GPU setups** where you want to leverage distributed resources for the VAE stage
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
|
||||
VAE Patch Parallelism uses two strategies based on image size:
|
||||
|
||||
| Strategy | Use Case | How It Works | Overlap Handling | Output Quality |
|
||||
|----------|----------|--------------|------------------|----------------|
|
||||
| **Tiled Decode** | Large images (triggers VAE tiling) | Distributes existing VAE tiling computation across ranks. Each rank decodes a subset of overlapping tiles. | Uses VAE's native `blend_v` and `blend_h` functions to seamlessly merge overlapping regions | Bit-identical (same logic as single-GPU tiling) |
|
||||
| **Patch Decode** | Small images (no VAE tiling) | Splits latent into spatial patches with halos. Each rank decodes one patch with boundary context. | Halo regions provide edge context; core regions are directly stitched without blending | Near-identical (diff < 0.5%, visually imperceptible) |
|
||||
|
||||
|
||||
VAE Patch Parallelism **reuses the DiT process group** (`dit_group`) and does not initialize a separate ProcessGroup. This means:
|
||||
|
||||
- **Shared ranks**: VAE patch parallelism uses the same GPU ranks as DiT parallelism (Tensor Parallel, Sequence Parallel, etc.)
|
||||
- **Combined usage**: VAE patch parallelism is typically used together with other parallelism methods
|
||||
- **Configuration alignment**: The `vae_patch_parallel_size` should be no greater than the size of your DiT process group
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simplest working example:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
# TP=2 for DiT, VAE patch parallel also uses these 2 GPUs
|
||||
omni = Omni(
|
||||
model="Tongyi-MAI/Z-Image-Turbo",
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
tensor_parallel_size=2, # Enable tensor parallelism for DiT
|
||||
vae_patch_parallel_size=2, # Enable VAE patch parallelism
|
||||
),
|
||||
vae_use_tiling=True, # Required for VAE patch parallelism
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
"a futuristic city at sunset, high resolution, 8k",
|
||||
OmniDiffusionSamplingParams(
|
||||
num_inference_steps=9,
|
||||
height=1152, # High resolution benefits from VAE patch parallel
|
||||
width=1152,
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Script
|
||||
|
||||
### Offline Inference
|
||||
|
||||
Use Python script under `examples/offline_inference/text_to_image/`:
|
||||
|
||||
```bash
|
||||
# Text-to-Image with Z-Image
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Tongyi-MAI/Z-Image-Turbo \
|
||||
--prompt "a futuristic city at sunset" \
|
||||
--height 1152 \
|
||||
--width 1152 \
|
||||
--tensor-parallel-size 2 \
|
||||
--vae-patch-parallel-size 2 \
|
||||
--vae-use-tiling
|
||||
```
|
||||
|
||||
### Online Serving
|
||||
|
||||
You can enable VAE patch parallelism in online serving via `--vae-patch-parallel-size`:
|
||||
|
||||
```bash
|
||||
# Text-to-Image with Z-Image, TP=2 + VAE patch parallel=2
|
||||
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091 \
|
||||
--tensor-parallel-size 2 \
|
||||
--vae-patch-parallel-size 2 \
|
||||
--vae-use-tiling
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
In `DiffusionParallelConfig`:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `vae_patch_parallel_size` | int | 1 | Number of GPUs for VAE patch/tile parallelism. Set to 2 or higher to enable. Should typically match `tensor_parallel_size` as they share the same process group. |
|
||||
|
||||
Additional requirements:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `vae_use_tiling` | bool | False | Must be set to `True` when using VAE patch parallelism. |
|
||||
|
||||
!!! note "Automatic VAE Tiling"
|
||||
When `vae_patch_parallel_size > 1` and the model has a distributed VAE (`DistributedVaeMixin`), the system automatically sets `vae_use_tiling=True` if not already enabled.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use
|
||||
|
||||
**Good for:**
|
||||
|
||||
- High-resolution image generation and long video generation
|
||||
- Memory-constrained setups where VAE decode causes OOM
|
||||
- Multi-GPU environments
|
||||
|
||||
**Not for:**
|
||||
|
||||
- Low-resolution images/videos where VAE decode is not a bottleneck
|
||||
- Single GPU setups should use vae tiling decode, but not parallel vae tiling decode
|
||||
- Models that do not support vae patch parallel
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issue 1: Model Not Support VAE Patch Parallel
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
WARNING: vae_patch_parallel_size=2 is set but VAE patch parallelism is NOT enabled for xxxPipeline; ignoring.
|
||||
```
|
||||
|
||||
**Root Cause**: VAE Patch Parallelism requires the model's VAE to implement `DistributedVaeMixin`. At startup, `vllm_omni/diffusion/registry.py` checks whether the instantiated pipeline has a `.vae` attribute that is an instance of `DistributedVaeMixin`. If it does not, the setting is silently ignored:
|
||||
|
||||
```python
|
||||
vae_pp_size = od_config.parallel_config.vae_patch_parallel_size
|
||||
is_distributed_vae = hasattr(model, "vae") and isinstance(model.vae, DistributedVaeMixin)
|
||||
if vae_pp_size > 1 and not is_distributed_vae:
|
||||
logger.warning(
|
||||
"vae_patch_parallel_size=%d is set but VAE patch parallelism is NOT enabled for %s; ignoring.",
|
||||
vae_pp_size,
|
||||
od_config.model_class_name,
|
||||
)
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Use a supported model** (recommended): check [Supported Models](../../diffusion_features.md#supported-models) for the VAE-Patch-Parallel column.
|
||||
|
||||
2. To add support for a new model, implement `DistributedVaeMixin` on its VAE class (contributions are welcome).
|
||||
|
||||
|
||||
### Common Issue 2: `vae_patch_parallel_size` Exceeds DiT Process Group Size
|
||||
|
||||
**Symptoms**: Shows warning message, and vae patch parallel size is resized to DiT process group size
|
||||
|
||||
**Root Cause**: VAE Patch Parallelism reuses the DiT process group.
|
||||
|
||||
**Recommendation**: Always set `vae_patch_parallel_size` to be no greater than your DiT process group size.
|
||||
|
||||
Note that the size of DiT process group size equals to:
|
||||
```text
|
||||
dit_parallel_size = data_parallel_size
|
||||
× cfg_parallel_size
|
||||
× sequence_parallel_size
|
||||
× pipeline_parallel_size
|
||||
× tensor_parallel_size
|
||||
|
||||
```
|
||||
_sequence_parallel_size = ulysses_degree × ring_degree_
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
1. ✅ **Enable VAE Patch Parallelism** - Set `vae_patch_parallel_size`， `vae_use_tiling=True` in `DiffusionParallelConfig` to reduce VAE decode peak memory
|
||||
2. ✅ **Use Long Sequence** - VAE patch parallelism benefits are most apparent at long sequence decoding
|
||||
3. ✅ **Combine with other parallelism methods** - Suggest to use together with Tensor Parallel or CFG-Parallel for maximum memory savings
|
||||
Reference in new issue
Block a user