[Docs] Refactor the attention backend docs/skill (#3475)

Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Canlin GuoandHongsheng Liu authored and GitHub committed 2026-05-11 09:13:07 +08:00
1 parent 67e0e10c85
commit ef7f2f1cd2
6 files changed
+186 -66

No files matched your search

@@ -115,17 +115,20 @@ class YourAttentionBlock(nn.Module):
from vllm_omni.diffusion.attention.layer import Attention
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
class YourAttentionBlock(nn.Module):
class YourSelfAttentionBlock(nn.Module):
def __init__(self, ...):
super().__init__()
# Initialize vLLM-Omni's Attention layer
# Initialize vLLM-Omni's Attention layer.
# `role` lets users target this site with --diffusion-attention-config
# (e.g. --diffusion-attention-config.per_role.self.backend SAGE_ATTN).
self.attn = Attention(
num_heads=self.num_heads,
head_size=self.head_dim,
softmax_scale=1.0 / (self.head_dim ** 0.5),
causal=False, # Diffusion models typically use bidirectional attention
num_kv_heads=self.num_kv_heads,
role="self",
)
def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, ...):
@@ -133,7 +136,6 @@ class YourAttentionBlock(nn.Module):
# Create attention metadata
attn_metadata = AttentionMetadata(attn_mask=attention_mask)
hidden_states = self.attn(query, key, value, attn_metadata=attn_metadata)
```
**Key Points:**
@@ -141,8 +143,35 @@ class YourAttentionBlock(nn.Module):
- **Attention layer initialization:** Done in `__init__`, not per-forward
- **Tensor shapes:** vLLM-Omni `Attention` expects QKV to have `[B, seq, num_heads, head_dim]` shape
- **AttentionMetadata:** Wraps attention mask and other metadata
- **Role:** Tag every `Attention` site with a `role` string so users can configure backends per role (see below)
**Attention backends:** vLLM-Omni automatically selects the attention backend given the environmental variable `DIFFUSION_ATTENTION_BACKEND`. The default attention backend is `FLASH_ATTN` for diffusion models.
**Declaring attention roles**
The `role` argument is a free-form string that identifies this attention site. Users can match it from `--diffusion-attention-config.per_role.<role>.*` to swap backends without touching model code. Two conventions cover the common cases:
| Convention | When to use | Example |
|---|---|---|
| `"self"` | Q/K/V come from the same hidden state | DiT self-attention block |
| `"cross"` | K/V come from a separate `encoder_hidden_states` | Text-conditioned cross-attention |
For multi-modal or unusual sites, use a dot-namespaced role and pair it with `role_category` so it can fall back to the generic config when nothing model-specific is set:
```python
# A model-specific cross-attention site that user config can target
# either as 'mymodel.audio_to_video' (exact) or as 'cross' (category fallback).
self.audio_to_video_attn = Attention(
num_heads=self.num_heads,
head_size=self.head_dim,
softmax_scale=1.0 / (self.head_dim ** 0.5),
causal=False,
role="mymodel.audio_to_video",
role_category="cross",
)
```
For cross-attention sites whose K/V are replicated across ranks (e.g. text encoder output), pass `skip_sequence_parallel=True` to opt this layer out of sequence-parallel sharding.
**Attention backends:** When the user does not configure a backend, vLLM-Omni asks the current platform for its default (typically `FLASH_ATTN` on CUDA when available). Users override the default via `--diffusion-attention-backend`, the `DIFFUSION_ATTENTION_BACKEND` env var, or finer-grained `--diffusion-attention-config.per_role.*` flags. See [Diffusion Attention Backends](../../user_guide/diffusion/attention_backends.md) for the full configuration surface.
#### 1.3: Replace Imports and Utilities