104 lines
3.4 KiB
Markdown
104 lines
3.4 KiB
Markdown
# TRTLLM Attention
|
|
|
|
`TRTLLM_ATTN` runs FlashInfer's trtllm-gen FMHA kernels. It supports dense
|
|
BF16 attention plus optional Skip-Softmax sparsity or SAGE quantization.
|
|
|
|
For common selection and per-role configuration, see the
|
|
[attention backend overview](../attention_backends.md).
|
|
|
|
## Requirements
|
|
|
|
Selecting `TRTLLM_ATTN` requires:
|
|
|
|
- datacenter Blackwell, sm_100 or sm_103;
|
|
- `head_dim=128`;
|
|
- FlashInfer; and
|
|
- a model-declared packed or mask-free compatible path.
|
|
|
|
It is not supported on workstation Blackwell (sm_120/sm_121). An incompatible
|
|
explicit selection raises instead of silently changing the backend.
|
|
|
|
## Dense mode
|
|
|
|
Select `TRTLLM_ATTN` without `skip_softmax` or `quant` to run dense BF16:
|
|
|
|
```bash
|
|
vllm-omni serve <model> --diffusion-attention-backend TRTLLM_ATTN
|
|
```
|
|
|
|
## Skip-Softmax
|
|
|
|
Skip-Softmax trades some fidelity for speed by skipping low-contribution
|
|
attention work. See the [feature design](../../../design/feature/skip_softmax.md)
|
|
for the algorithm and implementation contract.
|
|
|
|
| Key | Valid values | Meaning |
|
|
| --- | --- | --- |
|
|
| `target_sparsity` | finite, `[0, 1]` | Point on a model-calibrated curve |
|
|
| `threshold` | finite, `>= 0` | Direct threshold; mutually exclusive with `target_sparsity` |
|
|
| `disabled_until_timestep` | finite, `[0, 1]` | Keep early high-noise steps dense until normalized `t <= D` |
|
|
|
|
```bash
|
|
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
|
--diffusion-attention-config '{"default":{"backend":"TRTLLM_ATTN",\
|
|
"skip_softmax":{"target_sparsity":0.65,"disabled_until_timestep":0.86}}}'
|
|
```
|
|
|
|
```python
|
|
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, SkipSoftmaxSpec
|
|
|
|
config = AttentionConfig(
|
|
default=AttentionSpec(
|
|
backend="TRTLLM_ATTN",
|
|
skip_softmax=SkipSoftmaxSpec(
|
|
target_sparsity=0.65,
|
|
disabled_until_timestep=0.86,
|
|
),
|
|
),
|
|
)
|
|
```
|
|
|
|
Start with `target_sparsity=0.65` and `disabled_until_timestep=0.86`. Increase
|
|
sparsity only after comparing quality against dense output at the same seed.
|
|
The benefit grows with sequence length because only attention is accelerated.
|
|
|
|
## SAGE quantization
|
|
|
|
TRTLLM SAGE quantizes Q/K per block and V per channel so both attention
|
|
matrix multiplications use low precision.
|
|
|
|
| Key | Valid values | Meaning |
|
|
| --- | --- | --- |
|
|
| `dtype_qk` | `int8`, `fp8_e4m3` | Q/K quantization dtype; absent means dense |
|
|
| `q_block_size` | `1`, `4`, `16` | Q scale block size; default `1` |
|
|
| `k_block_size` | `1`, `4`, `16` | K scale block size; default `16` |
|
|
|
|
V is always quantized per channel to FP8 E4M3 and K-smoothing is internal.
|
|
Only block sizes with compiled kernels are accepted.
|
|
|
|
```bash
|
|
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
|
--diffusion-attention-config '{"default":{"backend":"TRTLLM_ATTN",\
|
|
"quant":{"dtype_qk":"fp8_e4m3","q_block_size":1,"k_block_size":16}}}'
|
|
```
|
|
|
|
```python
|
|
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, AttnQuantSpec
|
|
|
|
config = AttentionConfig(
|
|
default=AttentionSpec(
|
|
backend="TRTLLM_ATTN",
|
|
quant=AttnQuantSpec(
|
|
dtype_qk="fp8_e4m3",
|
|
q_block_size=1,
|
|
k_block_size=16,
|
|
),
|
|
),
|
|
)
|
|
```
|
|
|
|
This path requires FlashInfer 0.6.16rc1 or newer. FP8 Q/K kernels are
|
|
available on sm_100 and sm_103; INT8 Q/K kernels are compiled for sm_100 only.
|
|
The shared `AttnQuantSpec` is backend typed: values intended for
|
|
`FLASHINFER_ATTN` are rejected here and vice versa.
|