[Docs] Refactor the attention backend docs/skill (#3475)
Signed-off-by: gcanlin <canlinguosdu@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
67e0e10c85
commit
ef7f2f1cd2
6 files changed
+186
-66
No files matched your search
@@ -115,17 +115,20 @@ class YourAttentionBlock(nn.Module):
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
|
||||
|
||||
class YourAttentionBlock(nn.Module):
|
||||
class YourSelfAttentionBlock(nn.Module):
|
||||
def __init__(self, ...):
|
||||
super().__init__()
|
||||
|
||||
# Initialize vLLM-Omni's Attention layer
|
||||
# Initialize vLLM-Omni's Attention layer.
|
||||
# `role` lets users target this site with --diffusion-attention-config
|
||||
# (e.g. --diffusion-attention-config.per_role.self.backend SAGE_ATTN).
|
||||
self.attn = Attention(
|
||||
num_heads=self.num_heads,
|
||||
head_size=self.head_dim,
|
||||
softmax_scale=1.0 / (self.head_dim ** 0.5),
|
||||
causal=False, # Diffusion models typically use bidirectional attention
|
||||
num_kv_heads=self.num_kv_heads,
|
||||
role="self",
|
||||
)
|
||||
|
||||
def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, ...):
|
||||
@@ -133,7 +136,6 @@ class YourAttentionBlock(nn.Module):
|
||||
# Create attention metadata
|
||||
attn_metadata = AttentionMetadata(attn_mask=attention_mask)
|
||||
hidden_states = self.attn(query, key, value, attn_metadata=attn_metadata)
|
||||
|
||||
```
|
||||
|
||||
**Key Points:**
|
||||
@@ -141,8 +143,35 @@ class YourAttentionBlock(nn.Module):
|
||||
- **Attention layer initialization:** Done in `__init__`, not per-forward
|
||||
- **Tensor shapes:** vLLM-Omni `Attention` expects QKV to have `[B, seq, num_heads, head_dim]` shape
|
||||
- **AttentionMetadata:** Wraps attention mask and other metadata
|
||||
- **Role:** Tag every `Attention` site with a `role` string so users can configure backends per role (see below)
|
||||
|
||||
**Attention backends:** vLLM-Omni automatically selects the attention backend given the environmental variable `DIFFUSION_ATTENTION_BACKEND`. The default attention backend is `FLASH_ATTN` for diffusion models.
|
||||
**Declaring attention roles**
|
||||
|
||||
The `role` argument is a free-form string that identifies this attention site. Users can match it from `--diffusion-attention-config.per_role.<role>.*` to swap backends without touching model code. Two conventions cover the common cases:
|
||||
|
||||
| Convention | When to use | Example |
|
||||
|---|---|---|
|
||||
| `"self"` | Q/K/V come from the same hidden state | DiT self-attention block |
|
||||
| `"cross"` | K/V come from a separate `encoder_hidden_states` | Text-conditioned cross-attention |
|
||||
|
||||
For multi-modal or unusual sites, use a dot-namespaced role and pair it with `role_category` so it can fall back to the generic config when nothing model-specific is set:
|
||||
|
||||
```python
|
||||
# A model-specific cross-attention site that user config can target
|
||||
# either as 'mymodel.audio_to_video' (exact) or as 'cross' (category fallback).
|
||||
self.audio_to_video_attn = Attention(
|
||||
num_heads=self.num_heads,
|
||||
head_size=self.head_dim,
|
||||
softmax_scale=1.0 / (self.head_dim ** 0.5),
|
||||
causal=False,
|
||||
role="mymodel.audio_to_video",
|
||||
role_category="cross",
|
||||
)
|
||||
```
|
||||
|
||||
For cross-attention sites whose K/V are replicated across ranks (e.g. text encoder output), pass `skip_sequence_parallel=True` to opt this layer out of sequence-parallel sharding.
|
||||
|
||||
**Attention backends:** When the user does not configure a backend, vLLM-Omni asks the current platform for its default (typically `FLASH_ATTN` on CUDA when available). Users override the default via `--diffusion-attention-backend`, the `DIFFUSION_ATTENTION_BACKEND` env var, or finer-grained `--diffusion-attention-config.per_role.*` flags. See [Diffusion Attention Backends](../../user_guide/diffusion/attention_backends.md) for the full configuration surface.
|
||||
|
||||
#### 1.3: Replace Imports and Utilities
|
||||
|
||||
|
||||
Reference in new issue
Block a user