[CI] Select mirror_hardwares presets via MIRROR_HW (H100/B200) (#5543)
Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: lishunyang12 <lishunyang12@163.com> Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com> Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> Co-authored-by: lishunyang12 <lishunyang12@163.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Bo Li <22713281+bobboli@users.noreply.github.com>
This commit is contained in:
77 files changed
+3470
-1077
No files matched your search
@@ -21,10 +21,11 @@ vllm-omni serve <model> --diffusion-attention-backend TORCH_SDPA
|
||||
## `FLASH_ATTN`
|
||||
|
||||
`FLASH_ATTN` uses the installed FlashAttention implementation. On Blackwell it
|
||||
prefers the CuTe-based FlashAttention 4 path and falls back to
|
||||
FlashAttention 3 or 2 when that path is unavailable. On Hopper, Ada, and
|
||||
Ampere it is the preferred automatic route when a compatible package is
|
||||
installed.
|
||||
is FA4 only (`flash_attn.cute` / `vllm-omni[fa4]`). If FA4 is unavailable,
|
||||
explicit `FLASH_ATTN` is rejected and automatic selection continues to another
|
||||
compatible Blackwell backend. Hopper-only FA2/FA3 wheels are not used, even if
|
||||
they import. On Hopper, Ada, and Ampere it is the preferred automatic route
|
||||
when a compatible package is installed.
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> --diffusion-attention-backend FLASH_ATTN
|
||||
@@ -39,8 +40,10 @@ pip install 'vllm-omni[fa4]'
|
||||
```
|
||||
|
||||
Version `4.0.0b18` is required; earlier beta wheels had known JIT failures on
|
||||
Blackwell. If the CuTe path is unavailable, the backend falls back to the
|
||||
compatible FlashAttention 3 or 2 path.
|
||||
Blackwell. On Blackwell, `FLASH_ATTN` is FA4 only: Hopper-only FA2/FA3
|
||||
wheels are not used, even if they import. If FA4 is missing, explicit
|
||||
`FLASH_ATTN` is rejected and automatic selection continues to other
|
||||
Blackwell backends.
|
||||
|
||||
## `TRTLLM_ATTN`
|
||||
|
||||
@@ -89,6 +92,13 @@ DIFFUSION_ATTENTION_BACKEND=FLASHINFER_ATTN \
|
||||
option on CUDA platforms and an automatic Blackwell fallback when FlashInfer
|
||||
is installed but cuDNN is too old for `CUDNN_ATTN`.
|
||||
|
||||
On Blackwell, `auto` resolves to FlashInfer cute-dsl, which cannot run a
|
||||
nontrivial custom mask. Automatic selection may fall back to SDPA for those
|
||||
masks. An explicit `FLASHINFER_ATTN` selection does not: sequence-parallel
|
||||
auto-padding that would create a padding mask is rejected at capability
|
||||
preflight. Use `TORCH_SDPA`, pin `quant.flashinfer_backend` to `fa2`/`fa3`
|
||||
where those kernels exist, or choose a mask-capable backend.
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> --diffusion-attention-backend FLASHINFER_ATTN
|
||||
```
|
||||
|
||||
@@ -15,9 +15,9 @@ implementations to reduce numerical drift and sampling divergence.
|
||||
pip install kernels==0.14.1
|
||||
```
|
||||
|
||||
If `kernels` is unavailable, vLLM-Omni warns and falls back to the
|
||||
corresponding local FlashAttention implementation. On CUDA devices older than
|
||||
Hopper, `FLASH_ATTN_3_HUB` falls back to `FLASH_ATTN_HUB`.
|
||||
If `kernels` is unavailable, an explicit `FLASH_ATTN_HUB` or
|
||||
`FLASH_ATTN_3_HUB` selection raises. Automatic / `FLASH_ATTN` selection in the
|
||||
Diffusers adapter may still walk hub then local FlashAttention backends.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
Reference in new issue
Block a user