[Diffusion]: Diffusion Ulysses-Sequence-Parallelism support (#189)
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
This commit is contained in:
1 parent
f785a5d401
commit
85f0c1177f
21 files changed
+3513
-61
No files matched your search
@@ -0,0 +1,211 @@
|
||||
# Cache-DiT Acceleration Guide
|
||||
|
||||
This guide explains how to use cache-dit acceleration in vLLM-Omni to speed up diffusion model inference.
|
||||
|
||||
## Overview
|
||||
|
||||
Cache-dit is a library that accelerates diffusion transformer models through intelligent caching mechanisms. It supports multiple acceleration techniques that can be combined for optimal performance:
|
||||
|
||||
- **DBCache**: Dual Block Cache for reducing redundant computations
|
||||
- **TaylorSeer**: Taylor expansion-based forecasting for faster inference
|
||||
- **SCM**: Step Computation Masking for selective step computation
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Enable cache-dit acceleration by simply setting `cache_backend="cache_dit"`. Cache-dit will use its recommended default parameters:
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
# Simplest way: just enable cache-dit with default parameters
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
cache_backend="cache_dit",
|
||||
)
|
||||
|
||||
images = omni.generate(
|
||||
"a beautiful landscape",
|
||||
num_inference_steps=50,
|
||||
)
|
||||
```
|
||||
|
||||
**Default Parameters**: When `cache_config` is not provided, cache-dit uses optimized default values. See the [Configuration Reference](#configuration-reference) section for a complete list of all parameters and their default values.
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
To customize cache-dit settings, provide a `cache_config` dictionary, for example:
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
cache_backend="cache_dit",
|
||||
cache_config={
|
||||
"Fn_compute_blocks": 1,
|
||||
"Bn_compute_blocks": 0,
|
||||
"max_warmup_steps": 4,
|
||||
"residual_diff_threshold": 0.12,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Acceleration Methods
|
||||
|
||||
For comprehensive illustration, please view cache-dit [User_Guide](https://github.com/vipshop/cache-dit/blob/main/docs/User_Guide.md)
|
||||
|
||||
### 1. DBCache (Dual Block Cache)
|
||||
|
||||
DBCache intelligently caches intermediate transformer block outputs when the residual differences between consecutive steps are small, reducing redundant computations without sacrificing quality.
|
||||
|
||||
**Key Parameters**:
|
||||
|
||||
- `Fn_compute_blocks` (int, default: 1): Number of **first n** transformer blocks used to compute stable feature differences. Higher values provide more accurate caching decisions but increase computation.
|
||||
- `Bn_compute_blocks` (int, default: 0): Number of **last n** transformer blocks used for additional fusion. These blocks act as an auto-scaler for approximate hidden states.
|
||||
- `max_warmup_steps` (int, default: 4): Number of initial steps where caching is disabled to ensure the model learns sufficient features before caching begins. Optimized for few-step distilled models.
|
||||
- `residual_diff_threshold` (float, default: 0.24): Threshold for residual difference. Higher values lead to faster performance but may reduce precision. Default uses a relatively higher threshold for more aggressive caching.
|
||||
- `max_cached_steps` (int, default: -1): Maximum number of cached steps. Set to -1 for unlimited caching.
|
||||
- `max_continuous_cached_steps` (int, default: 3): Maximum number of consecutive cached steps. Limits consecutive caching to prevent precision degradation.
|
||||
|
||||
**Example Configuration**:
|
||||
|
||||
```python
|
||||
cache_config={
|
||||
"Fn_compute_blocks": 8, # Use first 8 blocks for difference computation
|
||||
"Bn_compute_blocks": 0, # No additional fusion blocks
|
||||
"max_warmup_steps": 8, # Cache after 8 warmup steps
|
||||
"residual_diff_threshold": 0.12, # Higher threshold for faster inference
|
||||
"max_cached_steps": -1, # No limit on cached steps
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Tips**:
|
||||
|
||||
- Default `Fn_compute_blocks=1` works well for most cases. Increase to 8-12 for larger models or when more accuracy is needed
|
||||
- Increase `residual_diff_threshold` (e.g., 0.12-0.15) for faster inference with slight quality trade-off, or decrease from default 0.24 for higher quality
|
||||
- Default `max_warmup_steps=4` is optimized for few-step models. Increase to 6-8 for more steps if needed
|
||||
|
||||
### 2. TaylorSeer
|
||||
|
||||
TaylorSeer uses Taylor expansion to forecast future hidden states, allowing the model to skip some computation steps while maintaining quality.
|
||||
|
||||
**Key Parameters**:
|
||||
|
||||
- `enable_taylorseer` (bool, default: False): Enable TaylorSeer acceleration
|
||||
- `taylorseer_order` (int, default: 1): Order of Taylor expansion. Higher orders provide better accuracy but require more computation.
|
||||
|
||||
**Example Configuration**:
|
||||
|
||||
```python
|
||||
cache_config={
|
||||
"enable_taylorseer": True,
|
||||
"taylorseer_order": 1, # First-order Taylor expansion
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Tips**:
|
||||
|
||||
- Use `taylorseer_order=1` for most cases (good balance of speed and quality)
|
||||
- Combine with DBCache for maximum acceleration
|
||||
- Higher orders (2-3) may improve quality but reduce speed gains
|
||||
|
||||
### 3. SCM (Step Computation Masking)
|
||||
|
||||
SCM allows you to specify which steps must be computed and which can use cached results, similar to LeMiCa/EasyCache style acceleration.
|
||||
|
||||
**Key Parameters**:
|
||||
|
||||
- `scm_steps_mask_policy` (str | None, default: None): Predefined mask policy. Options:
|
||||
- `None`: SCM disabled (default)
|
||||
- `"slow"`: More compute steps, higher quality (18 compute steps out of 28)
|
||||
- `"medium"`: Balanced (15 compute steps out of 28)
|
||||
- `"fast"`: More cache steps, faster inference (11 compute steps out of 28)
|
||||
- `"ultra"`: Maximum speed (8 compute steps out of 28)
|
||||
- `scm_steps_policy` (str, default: "dynamic"): Policy for cached steps:
|
||||
- `"dynamic"`: Use dynamic cache for masked steps (recommended)
|
||||
- `"static"`: Use static cache for masked steps
|
||||
|
||||
**Example Configuration**:
|
||||
|
||||
```python
|
||||
cache_config={
|
||||
"scm_steps_mask_policy": "medium", # Balanced speed/quality
|
||||
"scm_steps_policy": "dynamic", # Use dynamic cache
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Tips**:
|
||||
|
||||
- SCM is disabled by default (`scm_steps_mask_policy=None`). Enable it by setting a policy value if you need additional acceleration
|
||||
- Start with `"medium"` policy and adjust based on quality requirements
|
||||
- Use `"fast"` or `"ultra"` for maximum speed when quality can be slightly compromised
|
||||
- `"dynamic"` policy generally provides better quality than `"static"`
|
||||
- SCM mask is automatically regenerated when `num_inference_steps` changes during inference
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### DiffusionCacheConfig Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `Fn_compute_blocks` | int | 1 | First n blocks for difference computation (optimized for single-transformer models) |
|
||||
| `Bn_compute_blocks` | int | 0 | Last n blocks for fusion |
|
||||
| `max_warmup_steps` | int | 4 | Steps before caching starts (optimized for few-step distilled models) |
|
||||
| `max_cached_steps` | int | -1 | Max cached steps (-1 = unlimited) |
|
||||
| `max_continuous_cached_steps` | int | 3 | Max consecutive cached steps (prevents precision degradation) |
|
||||
| `residual_diff_threshold` | float | 0.24 | Residual difference threshold (higher for more aggressive caching) |
|
||||
| `num_inference_steps` | int \| None | None | Initial inference steps for SCM mask generation (optional, auto-refreshed during inference) |
|
||||
| `enable_taylorseer` | bool | False | Enable TaylorSeer acceleration (not suitable for few-step distilled models) |
|
||||
| `taylorseer_order` | int | 1 | Taylor expansion order |
|
||||
| `scm_steps_mask_policy` | str \| None | None | SCM mask policy (None, "slow", "medium", "fast", "ultra") |
|
||||
| `scm_steps_policy` | str | "dynamic" | SCM computation policy ("dynamic" or "static") |
|
||||
|
||||
## Example: Accelerate Text-to-Image Generation with CacheDiT
|
||||
|
||||
See `examples/offline_inference/text_to_image/text_to_image.py` for a complete working example with cache-dit acceleration.
|
||||
|
||||
```bash
|
||||
# Enable cache-dit with hybrid acceleration
|
||||
cd examples/offline_inference/text_to_image
|
||||
python text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--prompt "a cup of coffee on the table" \
|
||||
--cache_backend cache_dit \
|
||||
--num_inference_steps 50
|
||||
```
|
||||
|
||||
|
||||
The script uses cache-dit acceleration with a hybrid configuration combining DBCache, SCM, and TaylorSeer:
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
cache_backend="cache_dit",
|
||||
cache_config={
|
||||
# Scheme: Hybrid DBCache + SCM + TaylorSeer
|
||||
# DBCache
|
||||
"Fn_compute_blocks": 8,
|
||||
"Bn_compute_blocks": 0,
|
||||
"max_warmup_steps": 4,
|
||||
"residual_diff_threshold": 0.12,
|
||||
# TaylorSeer
|
||||
"enable_taylorseer": True,
|
||||
"taylorseer_order": 1,
|
||||
# SCM
|
||||
"scm_steps_mask_policy": "fast", # Set to None to disable SCM
|
||||
"scm_steps_policy": "dynamic",
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
You can customize the configuration by modifying the `cache_config` dictionary to use only specific methods (e.g., DBCache only, DBCache + SCM, etc.) based on your quality and speed requirements.
|
||||
|
||||
To test another model, you can modify `--model` with the target model identifier like `Tongyi-MAI/Z-Image-Turbo` and update `cache_config` according the model architecture (e.g., number of transformer blocks).
|
||||
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Cache-DiT User Guide](https://github.com/vipshop/cache-dit/blob/main/docs/User_Guide.md)
|
||||
- [Cache-DiT Benchmark](https://github.com/vipshop/cache-dit/tree/main/bench)
|
||||
- [DBCache Technical Details](https://github.com/vipshop/cache-dit/blob/main/docs/DBCache.md)
|
||||
@@ -0,0 +1,128 @@
|
||||
# Parallelism Acceleration Guide
|
||||
|
||||
This guide includes how to use parallelism methods in vLLM-Omni to speed up diffusion model inference as well as reduce the memory requirement on each device.
|
||||
|
||||
## Overview
|
||||
|
||||
The following parallelism methods are currently supported in vLLM-Omni:
|
||||
|
||||
1. DeepSpeed Ulysses Sequence Parallel (Ulysses-SP) ([paper](https://arxiv.org/pdf/2309.14509)): Ulysses-SP splits the input along the sequence dimension and uses all-to-all communication to allow each device to compute only a subset of attention heads.
|
||||
|
||||
|
||||
The following table shows which models are currently supported by parallelism method:
|
||||
|
||||
|
||||
| Model | Model Identifier | Ulysses-SP |
|
||||
|-------|-----------------|-----------|
|
||||
| **Qwen-Image** | `Qwen/Qwen-Image` | ✅ |
|
||||
| **Z-Image** | `Tongyi-MAI/Z-Image-Turbo` | ❌ |
|
||||
| **Qwen-Image-Edit** | `Qwen/Qwen-Image-Edit` | ✅ |
|
||||
|
||||
### Sequence Parallelism
|
||||
|
||||
#### Ulysses-SP
|
||||
|
||||
##### Quick Start
|
||||
|
||||
An example of using Ulysses-SP is shown below:
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
ulysses_degree = 2
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
parallel_config=DiffusionParallelConfig(ulysses_degree=2)
|
||||
)
|
||||
|
||||
outputs = omni.generate(prompt="A cat sitting on a windowsill", num_inference_steps=50, width=2048, height=2048)
|
||||
```
|
||||
|
||||
See `examples/offline_inference/text_to_image/text_to_image.py` for a complete working example.
|
||||
|
||||
##### Benchmarks
|
||||
!!! note "Benchmark Disclaimer"
|
||||
These benchmarks are provided for **general reference only**. The configurations shown use default or common parameter settings and have not been exhaustively optimized for maximum performance. Actual performance may vary based on:
|
||||
|
||||
- Specific model and use case
|
||||
- Hardware configuration
|
||||
- Careful parameter tuning
|
||||
- Different inference settings (e.g., number of steps, image resolution)
|
||||
|
||||
|
||||
To measure the parallelism methods, we run benchmarks with **Qwen/Qwen-Image** model generating images (**2048x2048** as long sequence input) with 50 inference steps. The hardware devices are NVIDIA H800 GPUs. `sdpa` is the attention backends.
|
||||
|
||||
| Configuration | Ulysses degree |Generation Time | Speedup |
|
||||
|---------------|----------------|---------|---------|
|
||||
| **Baseline (diffusers)** | - | 112.5s | 1.0x |
|
||||
| Ulysses-SP | 2 | 65.2s | 1.73x |
|
||||
| Ulysses-SP | 4 | 39.6s | 2.84x |
|
||||
| Ulysses-SP | 8 | 30.8s | 3.65x |
|
||||
|
||||
##### How to parallelize a new model
|
||||
|
||||
If a diffusion model has been deployed in vLLM-Omni and supports single-card inference, you can refer to the following instruction on how to parallelize this model with Ulysses-SP.
|
||||
|
||||
First, please edit the `TransformerModel`'s `forward` function in the `xxx_model_transformer.py` to make the inputs (image hidden states, positional embeddings, etc.) as chunks separated at the sequence dimension. Taking `qwen_image_transformer.py` as an example:
|
||||
|
||||
```diff
|
||||
class QwenImageTransformer2DModel(nn.Module):
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor = None,
|
||||
...
|
||||
):
|
||||
+ if self.parallel_config.sequence_parallel_size > 1:
|
||||
+ hidden_states = torch.chunk(hidden_states, get_sequence_parallel_world_size(), dim=-2)[
|
||||
+ get_sequence_parallel_rank()
|
||||
+ ]
|
||||
|
||||
hidden_states = self.img_in(hidden_states)
|
||||
|
||||
...
|
||||
image_rotary_emb = self.pos_embed(img_shapes, txt_seq_lens, device=hidden_states.device)
|
||||
|
||||
+ def get_rotary_emb_chunk(freqs):
|
||||
+ freqs = torch.chunk(freqs, get_sequence_parallel_world_size(), dim=0)[get_sequence_parallel_rank()]
|
||||
+ return freqs
|
||||
|
||||
+ if self.parallel_config.sequence_parallel_size > 1:
|
||||
+ img_freqs, txt_freqs = image_rotary_emb
|
||||
+ img_freqs = get_rotary_emb_chunk(img_freqs)
|
||||
+ image_rotary_emb = (img_freqs, txt_freqs)
|
||||
```
|
||||
|
||||
Next, at the end of the `forward` function, please call `get_sp_group().all_gather` to gather the chunked outputs across devices, and concatenate them at the sequence dimension.
|
||||
|
||||
|
||||
```diff
|
||||
class QwenImageTransformer2DModel(nn.Module):
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor = None,
|
||||
...
|
||||
):
|
||||
# Use only the image part (hidden_states) from the dual-stream blocks
|
||||
hidden_states = self.norm_out(hidden_states, temb)
|
||||
output = self.proj_out(hidden_states)
|
||||
|
||||
+ if self.parallel_config.sequence_parallel_size > 1:
|
||||
+ output = get_sp_group().all_gather(output, dim=-2)
|
||||
return Transformer2DModelOutput(sample=output)
|
||||
```
|
||||
|
||||
Finally, you can set the parallel configuration and pass it to `Omni` and start parallel inference with:
|
||||
```diff
|
||||
from vllm_omni import Omni
|
||||
+from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
ulysses_degree = 2
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
+ parallel_config=DiffusionParallelConfig(ulysses_degree=2)
|
||||
)
|
||||
|
||||
outputs = omni.generate(prompt="A cat sitting on a windowsill", num_inference_steps=50)
|
||||
```
|
||||
@@ -0,0 +1,99 @@
|
||||
# TeaCache Configuration Guide
|
||||
|
||||
TeaCache speeds up diffusion model inference by caching transformer computations when consecutive timesteps are similar. This typically provides **1.5x-2.0x speedup** with minimal quality loss.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Enable TeaCache by setting `cache_backend` to `"tea_cache"`:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
# Simple configuration - model_type is automatically extracted from pipeline.__class__.__name__
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
cache_backend="tea_cache",
|
||||
cache_config={
|
||||
"rel_l1_thresh": 0.2 # Optional, defaults to 0.2
|
||||
}
|
||||
)
|
||||
outputs = omni.generate(prompt="A cat sitting on a windowsill", num_inference_steps=50)
|
||||
```
|
||||
|
||||
### Using Environment Variable
|
||||
|
||||
You can also enable TeaCache via environment variable:
|
||||
|
||||
```bash
|
||||
export DIFFUSION_CACHE_BACKEND=tea_cache
|
||||
```
|
||||
|
||||
Then initialize without explicitly setting `cache_backend`:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
cache_config={"rel_l1_thresh": 0.2} # Optional
|
||||
)
|
||||
```
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
### `rel_l1_thresh` (float, default: `0.2`)
|
||||
|
||||
Controls the balance between speed and quality. Lower values prioritize quality, higher values prioritize speed.
|
||||
|
||||
**Recommended values:**
|
||||
|
||||
- `0.2` - **~1.5x speedup** with minimal quality loss (recommended)
|
||||
- `0.4` - **~1.8x speedup** with slight quality loss
|
||||
- `0.6` - **~2.0x speedup** with noticeable quality loss
|
||||
- `0.8` - **~2.25x speedup** with significant quality loss
|
||||
|
||||
## Examples
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
cache_backend="tea_cache",
|
||||
cache_config={"rel_l1_thresh": 0.2}
|
||||
)
|
||||
outputs = omni.generate(prompt="A cat sitting on a windowsill", num_inference_steps=50)
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
Start with the default `rel_l1_thresh=0.2` and adjust based on your needs:
|
||||
|
||||
- **Maximum quality**: Use `0.1-0.2`
|
||||
- **Balanced**: Use `0.2-0.4` (recommended)
|
||||
- **Maximum speed**: Use `0.6-0.8` (may reduce quality)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Quality Degradation
|
||||
|
||||
If you notice quality issues, lower the threshold:
|
||||
|
||||
```python
|
||||
cache_config={"rel_l1_thresh": 0.1} # More conservative caching
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
Currently supported models:
|
||||
|
||||
- **QwenImagePipeline**
|
||||
- Model identifier: `"Qwen/Qwen-Image"`
|
||||
- Example: `model="Qwen/Qwen-Image"`
|
||||
|
||||
Future support (extractors in development):
|
||||
|
||||
- **FluxPipeline** - Coming soon
|
||||
- **CogVideoXPipeline** - Coming soon
|
||||
Reference in new issue
Block a user