docs: split diffusion feature guides
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
8b0a6779d4
commit
7872663bf0
21 files changed
+1138
-801
No files matched your search
@@ -0,0 +1,103 @@
|
||||
# TRTLLM Attention
|
||||
|
||||
`TRTLLM_ATTN` runs FlashInfer's trtllm-gen FMHA kernels. It supports dense
|
||||
BF16 attention plus optional Skip-Softmax sparsity or SAGE quantization.
|
||||
|
||||
For common selection and per-role configuration, see the
|
||||
[attention backend overview](../attention_backends.md).
|
||||
|
||||
## Requirements
|
||||
|
||||
Selecting `TRTLLM_ATTN` requires:
|
||||
|
||||
- datacenter Blackwell, sm_100 or sm_103;
|
||||
- `head_dim=128`;
|
||||
- FlashInfer; and
|
||||
- a model-declared packed or mask-free compatible path.
|
||||
|
||||
It is not supported on workstation Blackwell (sm_120/sm_121). An incompatible
|
||||
explicit selection raises instead of silently changing the backend.
|
||||
|
||||
## Dense mode
|
||||
|
||||
Select `TRTLLM_ATTN` without `skip_softmax` or `quant` to run dense BF16:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> --diffusion-attention-backend TRTLLM_ATTN
|
||||
```
|
||||
|
||||
## Skip-Softmax
|
||||
|
||||
Skip-Softmax trades some fidelity for speed by skipping low-contribution
|
||||
attention work. See the [feature design](../../../design/feature/skip_softmax.md)
|
||||
for the algorithm and implementation contract.
|
||||
|
||||
| Key | Valid values | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `target_sparsity` | finite, `[0, 1]` | Point on a model-calibrated curve |
|
||||
| `threshold` | finite, `>= 0` | Direct threshold; mutually exclusive with `target_sparsity` |
|
||||
| `disabled_until_timestep` | finite, `[0, 1]` | Keep early high-noise steps dense until normalized `t <= D` |
|
||||
|
||||
```bash
|
||||
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--diffusion-attention-config '{"default":{"backend":"TRTLLM_ATTN",\
|
||||
"skip_softmax":{"target_sparsity":0.65,"disabled_until_timestep":0.86}}}'
|
||||
```
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, SkipSoftmaxSpec
|
||||
|
||||
config = AttentionConfig(
|
||||
default=AttentionSpec(
|
||||
backend="TRTLLM_ATTN",
|
||||
skip_softmax=SkipSoftmaxSpec(
|
||||
target_sparsity=0.65,
|
||||
disabled_until_timestep=0.86,
|
||||
),
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
Start with `target_sparsity=0.65` and `disabled_until_timestep=0.86`. Increase
|
||||
sparsity only after comparing quality against dense output at the same seed.
|
||||
The benefit grows with sequence length because only attention is accelerated.
|
||||
|
||||
## SAGE quantization
|
||||
|
||||
TRTLLM SAGE quantizes Q/K per block and V per channel so both attention
|
||||
matrix multiplications use low precision.
|
||||
|
||||
| Key | Valid values | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `dtype_qk` | `int8`, `fp8_e4m3` | Q/K quantization dtype; absent means dense |
|
||||
| `q_block_size` | `1`, `4`, `16` | Q scale block size; default `1` |
|
||||
| `k_block_size` | `1`, `4`, `16` | K scale block size; default `16` |
|
||||
|
||||
V is always quantized per channel to FP8 E4M3 and K-smoothing is internal.
|
||||
Only block sizes with compiled kernels are accepted.
|
||||
|
||||
```bash
|
||||
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--diffusion-attention-config '{"default":{"backend":"TRTLLM_ATTN",\
|
||||
"quant":{"dtype_qk":"fp8_e4m3","q_block_size":1,"k_block_size":16}}}'
|
||||
```
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, AttnQuantSpec
|
||||
|
||||
config = AttentionConfig(
|
||||
default=AttentionSpec(
|
||||
backend="TRTLLM_ATTN",
|
||||
quant=AttnQuantSpec(
|
||||
dtype_qk="fp8_e4m3",
|
||||
q_block_size=1,
|
||||
k_block_size=16,
|
||||
),
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
This path requires FlashInfer 0.6.16rc1 or newer. FP8 Q/K kernels are
|
||||
available on sm_100 and sm_103; INT8 Q/K kernels are compiled for sm_100 only.
|
||||
The shared `AttnQuantSpec` is backend typed: values intended for
|
||||
`FLASHINFER_ATTN` are rejected here and vice versa.
|
||||
Reference in new issue
Block a user