[Attention] Add trtllm diffusion attention backend with Skip-Softmax (#5283)
Signed-off-by: lishunyang12 <lishunyang03@gmail.com> Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
This commit is contained in:
1 parent
4dd1b47f47
commit
7762d664d8
24 files changed
+1512
-52
No files matched your search
@@ -561,7 +561,8 @@ class OmniServeCommand(CLISubcommand):
|
||||
help="Diffusion attention config. Accepts JSON or vLLM-style dotted flags. "
|
||||
"Examples: "
|
||||
"--diffusion-attention-config.default.backend FLASH_ATTN, "
|
||||
"--diffusion-attention-config.per_role.self.backend SPARSE_BLOCK, "
|
||||
"--diffusion-attention-config.default.backend TRTLLM_ATTN "
|
||||
"--diffusion-attention-config.default.skip_softmax.target_sparsity 0.5, "
|
||||
"--diffusion-attention-config.per_role.cross.backend SAGE_ATTN, "
|
||||
'--diffusion-attention-config \'{"default": {"backend": "FLASH_ATTN"}, '
|
||||
'"per_role": {"cross": {"backend": "SAGE_ATTN"}}}\'.',
|
||||
|
||||
Reference in new issue
Block a user