[Feature]: add Ulysses advanced_uaa mode (#1379)
Signed-off-by: dongbo910220 <1275604947@qq.com> Signed-off-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com> Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
This commit is contained in:
1 parent
4cfa9bf843
commit
50bd1961bf
17 files changed
+1064
-13
No files matched your search
@@ -112,6 +112,7 @@ steps:
|
||||
depends_on: upload-merge-pipeline
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_sequence_parallel.py
|
||||
- pytest -s -v tests/diffusion/distributed/test_ulysses_uaa_perf.py
|
||||
agents:
|
||||
queue: "gpu_4_queue" # g6.12xlarge instance on AWS, has 4 L4 GPU
|
||||
plugins:
|
||||
|
||||
@@ -7,6 +7,7 @@ This section describes how to add Sequence Parallel (SP) to a diffusion transfor
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [UAA Mode (Experimental)](#uaa-mode-experimental)
|
||||
- [Approach 1: Non-Intrusive `_sp_plan` (Recommended)](#approach-1-non-intrusive-_sp_plan-recommended)
|
||||
- [Approach 2: Intrusive Modification (For Complex Cases)](#approach-2-intrusive-modification-for-complex-cases)
|
||||
- [Testing](#testing)
|
||||
@@ -46,6 +47,41 @@ from vllm_omni.diffusion.distributed.sp_sharding import sp_shard, sp_gather
|
||||
|
||||
---
|
||||
|
||||
## UAA Mode (Experimental)
|
||||
|
||||
`ulysses_mode="advanced_uaa"` enables the experimental UAA ("Ulysses Anything Attention") path that was introduced to address the uneven-shape cases discussed in [issue #1324](https://github.com/vllm-project/vllm-omni/issues/1324).
|
||||
|
||||
Use it when plain Ulysses-SP would otherwise fail because:
|
||||
|
||||
- the local sequence shards are not evenly divisible after split hooks, or
|
||||
- the attention head count is not divisible by `ulysses_degree`.
|
||||
|
||||
### Design Summary
|
||||
|
||||
1. **Strict mode stays unchanged.**
|
||||
`ulysses_mode="strict"` keeps the original fast path and still requires divisible sequence/head shapes.
|
||||
|
||||
2. **UAA uses variable all-to-all split sizes for sequence shards.**
|
||||
Before the Ulysses Q/K/V exchange, each rank all-gathers its local sequence length and uses those lengths as `all_to_all_single(..., output_split_sizes=seq_lens)`. This lets Ulysses gather the full sequence even when each rank started with a different local shard length.
|
||||
|
||||
3. **UAA pads heads only inside the Ulysses exchange.**
|
||||
If `head_cnt % ulysses_degree != 0`, UAA pads the head dimension up to the next multiple of `ulysses_degree`, performs the forward/reverse all-to-all, then slices the temporary head padding away after the reverse exchange. The same rule is applied to joint attention tensors.
|
||||
|
||||
4. **Hybrid Ulysses + Ring is still shape-constrained.**
|
||||
Ring attention expects every rank in a ring group to exchange exactly the same post-Ulysses sequence shape. UAA therefore validates those shapes before entering the ring path and raises a clear error if ring peers disagree on `S_global`.
|
||||
|
||||
5. **Tiny scalar gathers stay out of TorchDynamo tracing.**
|
||||
`_all_gather_int()` is marked with `@torch.compiler.disable` so the scalar `item()` conversions used by UAA metadata collection do not get traced into `torch.compile`.
|
||||
|
||||
### UAA vs `auto_pad`
|
||||
|
||||
- `auto_pad=True` pads sequence tokens in `_sp_plan` and requires attention backends that support `attention_mask`.
|
||||
- `advanced_uaa` does not depend on mask-based token padding inside Ulysses attention. It is therefore a better fit for non-divisible head counts and uneven Ulysses shard sizes.
|
||||
- `auto_pad=True` remains incompatible with Ring attention because the ring backend does not consume `attention_mask`.
|
||||
- `advanced_uaa` is still experimental and hybrid mode remains limited by Ring's equal-shape requirement.
|
||||
|
||||
---
|
||||
|
||||
## Approach 1: Non-Intrusive `_sp_plan` (Recommended)
|
||||
|
||||
The `_sp_plan` mechanism allows SP **without modifying `forward()` logic**. The framework automatically registers hooks to shard inputs and gather outputs at module boundaries.
|
||||
@@ -364,9 +400,14 @@ _sp_plan = {
|
||||
|
||||
- **Sequence Length not divisible by sp_size:**
|
||||
|
||||
**Problem:** `SequenceParallelInput(auto_pad=False)` - auto_pad should be True to enable automatic sequence padding.
|
||||
**Problem:** strict Ulysses sequence parallel requires divisible shapes. If the shard length is uneven, or if the model head count is not divisible by `ulysses_degree`, the strict path will raise an error.
|
||||
|
||||
**Solution:** In `SequenceParallelInput`, set `auto_pad=True` and add attention mask support.
|
||||
**Solutions:**
|
||||
|
||||
1. Use `ulysses_mode="advanced_uaa"` for Ulysses-SP when you want the experimental uneven-shape path without relying on attention-mask padding.
|
||||
2. If the model already uses `_sp_plan` token padding and the attention backend supports `attention_mask`, set `auto_pad=True` and add attention-mask plumbing.
|
||||
|
||||
> **Experimental Feature:** `ulysses_mode="advanced_uaa"` is experimental. It is intended to relax Ulysses divisibility constraints, but hybrid Ulysses + Ring still requires equal post-Ulysses sequence lengths inside each ring group.
|
||||
|
||||
> **Experimental Feature:** `auto_pad=True` is an experimental feature and may be changed in the future. We plan to improve this solution to involve minimal changes to model files. More details are [here](https://github.com/vllm-project/vllm-omni/issues/1324).
|
||||
|
||||
|
||||
@@ -134,6 +134,11 @@ VAE patch parallelism automatically selects between two internal decode methods
|
||||
|
||||
#### Ulysses-SP
|
||||
|
||||
!!! note "Experimental UAA mode"
|
||||
`ulysses_mode="advanced_uaa"` is an experimental extension to Ulysses-SP. It allows Ulysses attention to handle non-divisible attention head counts and uneven per-rank sequence shards without relying on `attention_mask`-based token padding.
|
||||
|
||||
In hybrid Ulysses + Ring mode, Ring still requires every rank in the same ring group to observe the same post-Ulysses sequence length. If that condition is not met, vLLM-Omni raises a validation error instead of entering the ring kernel with inconsistent shapes.
|
||||
|
||||
##### Offline Inference
|
||||
|
||||
An example of offline inference script using [Ulysses-SP](https://arxiv.org/pdf/2309.14509) is shown below:
|
||||
@@ -156,6 +161,18 @@ outputs = omni.generate(
|
||||
|
||||
See `examples/offline_inference/text_to_image/text_to_image.py` for a complete working example.
|
||||
|
||||
To enable the experimental UAA mode explicitly:
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="Tongyi-MAI/Z-Image-Turbo",
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
ulysses_degree=4,
|
||||
ulysses_mode="advanced_uaa",
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
##### Online Serving
|
||||
|
||||
You can enable Ulysses-SP in online serving for diffusion models via `--usp`:
|
||||
@@ -163,6 +180,9 @@ You can enable Ulysses-SP in online serving for diffusion models via `--usp`:
|
||||
```bash
|
||||
# Text-to-image (requires >= 2 GPUs)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2
|
||||
|
||||
# Experimental UAA mode
|
||||
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091 --usp 4 --ulysses-mode advanced_uaa
|
||||
```
|
||||
|
||||
##### Benchmarks
|
||||
@@ -243,6 +263,9 @@ To measure the parallelism methods, we run benchmarks with **Qwen/Qwen-Image** m
|
||||
|
||||
You can combine both Ulysses-SP and Ring-Attention for larger scale parallelism. The total sequence parallel size equals `ulysses_degree × ring_degree`.
|
||||
|
||||
!!! note "Experimental UAA in hybrid mode"
|
||||
`ulysses_mode="advanced_uaa"` can also be used with hybrid Ulysses + Ring, but this does not remove Ring's shape requirement. Every rank in the same ring group must still have the same post-Ulysses sequence length.
|
||||
|
||||
##### Offline Inference
|
||||
|
||||
```python
|
||||
@@ -253,7 +276,11 @@ from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
# Hybrid: 2 Ulysses × 2 Ring = 4 GPUs total
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image",
|
||||
parallel_config=DiffusionParallelConfig(ulysses_degree=2, ring_degree=2)
|
||||
parallel_config=DiffusionParallelConfig(
|
||||
ulysses_degree=2,
|
||||
ring_degree=2,
|
||||
ulysses_mode="advanced_uaa",
|
||||
)
|
||||
)
|
||||
|
||||
outputs = omni.generate(
|
||||
@@ -266,7 +293,7 @@ outputs = omni.generate(
|
||||
|
||||
```bash
|
||||
# Text-to-image (requires >= 4 GPUs)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2 --ulysses-mode advanced_uaa
|
||||
```
|
||||
|
||||
##### Benchmarks
|
||||
|
||||
@@ -84,6 +84,9 @@ python text_to_image.py \
|
||||
| `--vae-use-slicing` | flag | off | Enable VAE slicing for memory optimization |
|
||||
| `--vae-use-tiling` | flag | off | Enable VAE tiling for memory optimization |
|
||||
| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel |
|
||||
| `--ulysses-degree` | int | `1` | Ulysses sequence parallel degree for multi-GPU inference |
|
||||
| `--ring-degree` | int | `1` | Ring sequence parallel degree for hybrid Ulysses + Ring inference |
|
||||
| `--ulysses-mode` | str | `"strict"` | Ulysses SP mode: `"strict"` or `"advanced_uaa"` |
|
||||
| `--enable-cpu-offload` | flag | off | Enable CPU offloading for diffusion models |
|
||||
| `--lora-path` | str | — | Path to PEFT LoRA adapter folder |
|
||||
| `--lora-scale` | float | `1.0` | Scale factor for LoRA weights |
|
||||
@@ -120,6 +123,20 @@ python text_to_image.py \
|
||||
|
||||
`Tongyi-MAI/Z-Image-Turbo` is a distilled version of Z-Image. Distilled diffusion models usually require less number of inference steps (4~9), and Classifier-Free Guidance (CFG) is usually NOT applied. Similar distilled models are `black-forest-labs/FLUX.2-klein-4B` and `black-forest-labs/FLUX.2-klein-9B`.
|
||||
|
||||
Hybrid Ulysses + Ring example (requires 4 GPUs):
|
||||
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
--model Tongyi-MAI/Z-Image-Turbo \
|
||||
--prompt "a cup of coffee on the table" \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--ulysses-mode advanced_uaa \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output outputs/coffee_hybrid.png
|
||||
```
|
||||
|
||||
### NextStep Models
|
||||
|
||||
NextStep-1.1 supports extra arguments for dual-level CFG control:
|
||||
|
||||
@@ -109,6 +109,13 @@ def parse_args() -> argparse.Namespace:
|
||||
default=1,
|
||||
help="Number of GPUs used for ulysses sequence parallelism.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ulysses-mode",
|
||||
type=str,
|
||||
default="strict",
|
||||
choices=["strict", "advanced_uaa"],
|
||||
help="Ulysses sequence-parallel mode: 'strict' (divisibility required) or 'advanced_uaa' (UAA).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ring-degree",
|
||||
type=int,
|
||||
@@ -272,6 +279,7 @@ def main():
|
||||
parallel_config = DiffusionParallelConfig(
|
||||
ulysses_degree=args.ulysses_degree,
|
||||
ring_degree=args.ring_degree,
|
||||
ulysses_mode=args.ulysses_mode,
|
||||
cfg_parallel_size=args.cfg_parallel_size,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
vae_patch_parallel_size=args.vae_patch_parallel_size,
|
||||
@@ -344,8 +352,10 @@ def main():
|
||||
print(f" Ignored layers: {ignored_layers}")
|
||||
print(
|
||||
f" Parallel configuration: tensor_parallel_size={args.tensor_parallel_size}, "
|
||||
f"ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, "
|
||||
f"vae_patch_parallel_size={args.vae_patch_parallel_size}, enable_expert_parallel={args.enable_expert_parallel}."
|
||||
f"ulysses_degree={args.ulysses_degree}, ulysses_mode={args.ulysses_mode}, "
|
||||
f"ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, "
|
||||
f"vae_patch_parallel_size={args.vae_patch_parallel_size}, "
|
||||
f"enable_expert_parallel={args.enable_expert_parallel}."
|
||||
)
|
||||
print(f" CPU offload: {args.enable_cpu_offload}")
|
||||
print(f" Image size: {args.width}x{args.height}")
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import socket
|
||||
import tempfile
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig, OmniDiffusionConfig
|
||||
from vllm_omni.diffusion.distributed.parallel_state import (
|
||||
destroy_distributed_env,
|
||||
init_distributed_environment,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
from vllm_omni.diffusion.forward_context import get_forward_context, set_forward_context
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
|
||||
def _find_free_port() -> int:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
return int(s.getsockname()[1])
|
||||
|
||||
|
||||
def _set_dist_env(*, rank: int, world_size: int, master_port: int) -> None:
|
||||
os.environ["RANK"] = str(rank)
|
||||
os.environ["LOCAL_RANK"] = str(rank)
|
||||
os.environ["WORLD_SIZE"] = str(world_size)
|
||||
os.environ["MASTER_ADDR"] = "127.0.0.1"
|
||||
os.environ["MASTER_PORT"] = str(master_port)
|
||||
|
||||
|
||||
def _run_attention_case(
|
||||
local_rank: int,
|
||||
world_size: int,
|
||||
master_port: int,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
num_heads: int,
|
||||
head_size: int,
|
||||
ulysses_degree: int,
|
||||
ulysses_mode: str,
|
||||
ring_degree: int = 1,
|
||||
split_sizes: list[int] | None = None,
|
||||
sdp_kernel_mode: str = "math",
|
||||
) -> None:
|
||||
device = torch.device(f"{current_omni_platform.device_type}:{local_rank}")
|
||||
current_omni_platform.set_device(device)
|
||||
|
||||
_set_dist_env(rank=local_rank, world_size=world_size, master_port=master_port)
|
||||
os.environ["DIFFUSION_ATTENTION_BACKEND"] = "TORCH_SDPA"
|
||||
|
||||
init_distributed_environment(world_size=world_size, rank=local_rank)
|
||||
initialize_model_parallel(
|
||||
data_parallel_size=1,
|
||||
cfg_parallel_size=1,
|
||||
sequence_parallel_size=world_size,
|
||||
ulysses_degree=ulysses_degree,
|
||||
ring_degree=ring_degree,
|
||||
tensor_parallel_size=1,
|
||||
pipeline_parallel_size=1,
|
||||
)
|
||||
|
||||
parallel_config = DiffusionParallelConfig(
|
||||
pipeline_parallel_size=1,
|
||||
data_parallel_size=1,
|
||||
tensor_parallel_size=1,
|
||||
sequence_parallel_size=world_size,
|
||||
ulysses_degree=ulysses_degree,
|
||||
ring_degree=ring_degree,
|
||||
cfg_parallel_size=1,
|
||||
ulysses_mode=ulysses_mode,
|
||||
)
|
||||
od_config = OmniDiffusionConfig(model="test", dtype=torch.float32, parallel_config=parallel_config)
|
||||
|
||||
with set_forward_context(omni_diffusion_config=od_config):
|
||||
attn = Attention(
|
||||
num_heads=num_heads,
|
||||
head_size=head_size,
|
||||
causal=False,
|
||||
softmax_scale=1.0 / (head_size**0.5),
|
||||
).to(device=device, dtype=torch.float32)
|
||||
|
||||
with np.load(input_file, allow_pickle=False) as payload:
|
||||
q_full = torch.from_numpy(payload["q"]).to(device=device)
|
||||
k_full = torch.from_numpy(payload["k"]).to(device=device)
|
||||
v_full = torch.from_numpy(payload["v"]).to(device=device)
|
||||
|
||||
if world_size == 1:
|
||||
q, k, v = q_full, k_full, v_full
|
||||
else:
|
||||
if split_sizes is None:
|
||||
# NOTE: torch.chunk may return fewer than `world_size` chunks for some
|
||||
# uneven lengths (e.g. seq_len=9, world_size=4 -> 3 chunks of len 3).
|
||||
# We need an exact `world_size`-way split to simulate uneven SP shards.
|
||||
q = torch.tensor_split(q_full, world_size, dim=1)[local_rank].contiguous()
|
||||
k = torch.tensor_split(k_full, world_size, dim=1)[local_rank].contiguous()
|
||||
v = torch.tensor_split(v_full, world_size, dim=1)[local_rank].contiguous()
|
||||
else:
|
||||
if len(split_sizes) != world_size:
|
||||
raise ValueError(f"split_sizes length ({len(split_sizes)}) must equal world_size ({world_size}).")
|
||||
if sum(int(x) for x in split_sizes) != q_full.shape[1]:
|
||||
raise ValueError(
|
||||
"split_sizes must sum to full seq_len "
|
||||
f"(got sum={sum(int(x) for x in split_sizes)}, seq_len={q_full.shape[1]})."
|
||||
)
|
||||
q = torch.split(q_full, split_sizes, dim=1)[local_rank].contiguous()
|
||||
k = torch.split(k_full, split_sizes, dim=1)[local_rank].contiguous()
|
||||
v = torch.split(v_full, split_sizes, dim=1)[local_rank].contiguous()
|
||||
# The Attention layer only enables SP communication when ForwardContext.sp_active is True.
|
||||
# In production this is managed by SequenceParallelSplitHook/GatherHook, but here we
|
||||
# shard manually for testing.
|
||||
get_forward_context()._sp_shard_depth = 1
|
||||
|
||||
if sdp_kernel_mode == "math":
|
||||
sdp_ctx = torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True)
|
||||
elif sdp_kernel_mode == "mem_efficient":
|
||||
sdp_ctx = torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=True, enable_math=False)
|
||||
else:
|
||||
raise ValueError(f"Invalid sdp_kernel_mode: {sdp_kernel_mode!r}")
|
||||
|
||||
with sdp_ctx:
|
||||
out_local = attn(q, k, v).contiguous()
|
||||
|
||||
if world_size == 1:
|
||||
out_full = out_local
|
||||
else:
|
||||
local_len = torch.tensor([out_local.shape[1]], device=device, dtype=torch.int64)
|
||||
gathered_lens = [torch.empty_like(local_len) for _ in range(world_size)]
|
||||
torch.distributed.all_gather(gathered_lens, local_len)
|
||||
lens = [int(t.item()) for t in gathered_lens]
|
||||
max_len = max(lens)
|
||||
|
||||
if out_local.shape[1] < max_len:
|
||||
pad = max_len - out_local.shape[1]
|
||||
out_local = torch.nn.functional.pad(out_local, (0, 0, 0, 0, 0, pad)).contiguous()
|
||||
|
||||
gathered = [torch.empty_like(out_local) for _ in range(world_size)]
|
||||
torch.distributed.all_gather(gathered, out_local)
|
||||
if local_rank == 0:
|
||||
out_full = torch.cat([t[:, : lens[i]].contiguous() for i, t in enumerate(gathered)], dim=1)
|
||||
else:
|
||||
out_full = None
|
||||
|
||||
if local_rank == 0:
|
||||
np.save(output_file, out_full.detach().cpu().numpy())
|
||||
|
||||
destroy_distributed_env()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"sp_world_size,seq_len,num_heads",
|
||||
[
|
||||
(2, 6, 3), # head_cnt not divisible by P=2
|
||||
(2, 5, 4), # seq_len not divisible by P=2
|
||||
(4, 9, 30), # Z-Image-like: head_cnt not divisible by P=4
|
||||
(4, 10, 8), # seq_len not divisible by P=4
|
||||
],
|
||||
)
|
||||
def test_ulysses_uaa_matches_baseline(sp_world_size: int, seq_len: int, num_heads: int) -> None:
|
||||
if current_omni_platform.get_device_count() < sp_world_size:
|
||||
pytest.skip(f"Test requires {sp_world_size} GPUs")
|
||||
|
||||
batch_size = 2
|
||||
head_size = 8
|
||||
|
||||
base_port = _find_free_port()
|
||||
sp_port = _find_free_port()
|
||||
while sp_port == base_port:
|
||||
sp_port = _find_free_port()
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".npz") as f_in:
|
||||
input_file = f_in.name
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".npy") as f_base:
|
||||
baseline_file = f_base.name
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".npy") as f_sp:
|
||||
sp_file = f_sp.name
|
||||
|
||||
try:
|
||||
torch.manual_seed(0)
|
||||
q = torch.randn(batch_size, seq_len, num_heads, head_size, dtype=torch.float32)
|
||||
k = torch.randn(batch_size, seq_len, num_heads, head_size, dtype=torch.float32)
|
||||
v = torch.randn(batch_size, seq_len, num_heads, head_size, dtype=torch.float32)
|
||||
np.savez(input_file, q=q.numpy(), k=k.numpy(), v=v.numpy())
|
||||
|
||||
# Baseline (no SP)
|
||||
torch.multiprocessing.spawn(
|
||||
_run_attention_case,
|
||||
args=(1, base_port, input_file, baseline_file, num_heads, head_size, 1, "strict"),
|
||||
nprocs=1,
|
||||
)
|
||||
|
||||
# SP (Ulysses-P with UAA)
|
||||
torch.multiprocessing.spawn(
|
||||
_run_attention_case,
|
||||
args=(
|
||||
sp_world_size,
|
||||
sp_port,
|
||||
input_file,
|
||||
sp_file,
|
||||
num_heads,
|
||||
head_size,
|
||||
sp_world_size,
|
||||
"advanced_uaa",
|
||||
),
|
||||
nprocs=sp_world_size,
|
||||
)
|
||||
|
||||
baseline = np.load(baseline_file, allow_pickle=False)
|
||||
sp = np.load(sp_file, allow_pickle=False)
|
||||
|
||||
baseline_t = torch.from_numpy(baseline)
|
||||
sp_t = torch.from_numpy(sp)
|
||||
assert baseline_t.shape == sp_t.shape
|
||||
torch.testing.assert_close(sp_t, baseline_t, atol=1e-5, rtol=1e-5)
|
||||
finally:
|
||||
for path in (input_file, baseline_file, sp_file):
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def test_ulysses_uaa_hybrid_ring_matches_baseline() -> None:
|
||||
sp_world_size = 4
|
||||
ulysses_degree = 2
|
||||
ring_degree = 2
|
||||
|
||||
if current_omni_platform.get_device_count() < sp_world_size:
|
||||
pytest.skip(f"Test requires {sp_world_size} GPUs")
|
||||
|
||||
batch_size = 2
|
||||
head_size = 8
|
||||
seq_len = 10
|
||||
num_heads = 3 # head_cnt not divisible by ulysses_degree=2 -> triggers head padding
|
||||
|
||||
# Ensure ring ranks see equal post-Ulysses seq_len:
|
||||
# rank0/1 -> 3+2=5, rank2/3 -> 3+2=5
|
||||
split_sizes = [3, 2, 3, 2]
|
||||
|
||||
base_port = _find_free_port()
|
||||
sp_port = _find_free_port()
|
||||
while sp_port == base_port:
|
||||
sp_port = _find_free_port()
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".npz") as f_in:
|
||||
input_file = f_in.name
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".npy") as f_base:
|
||||
baseline_file = f_base.name
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".npy") as f_sp:
|
||||
sp_file = f_sp.name
|
||||
|
||||
try:
|
||||
torch.manual_seed(0)
|
||||
q = torch.randn(batch_size, seq_len, num_heads, head_size, dtype=torch.float32)
|
||||
k = torch.randn(batch_size, seq_len, num_heads, head_size, dtype=torch.float32)
|
||||
v = torch.randn(batch_size, seq_len, num_heads, head_size, dtype=torch.float32)
|
||||
np.savez(input_file, q=q.numpy(), k=k.numpy(), v=v.numpy())
|
||||
|
||||
# Baseline (no SP)
|
||||
torch.multiprocessing.spawn(
|
||||
_run_attention_case,
|
||||
args=(1, base_port, input_file, baseline_file, num_heads, head_size, 1, "strict", 1, None, "mem_efficient"),
|
||||
nprocs=1,
|
||||
)
|
||||
|
||||
# Hybrid SP: Ulysses (P=2) + Ring (P=2) with advanced_uaa
|
||||
torch.multiprocessing.spawn(
|
||||
_run_attention_case,
|
||||
args=(
|
||||
sp_world_size,
|
||||
sp_port,
|
||||
input_file,
|
||||
sp_file,
|
||||
num_heads,
|
||||
head_size,
|
||||
ulysses_degree,
|
||||
"advanced_uaa",
|
||||
ring_degree,
|
||||
split_sizes,
|
||||
"mem_efficient",
|
||||
),
|
||||
nprocs=sp_world_size,
|
||||
)
|
||||
|
||||
baseline = np.load(baseline_file, allow_pickle=False)
|
||||
sp = np.load(sp_file, allow_pickle=False)
|
||||
|
||||
baseline_t = torch.from_numpy(baseline)
|
||||
sp_t = torch.from_numpy(sp)
|
||||
assert baseline_t.shape == sp_t.shape
|
||||
# Hybrid (Ulysses+Ring) typically has slightly larger numerical differences
|
||||
# than pure Ulysses due to different communication/reduction order and
|
||||
# the SDPA kernel path used by Ring attention. Use a looser tolerance to
|
||||
# keep the test stable across GPUs/kernels while still catching regressions.
|
||||
torch.testing.assert_close(sp_t, baseline_t, atol=5e-4, rtol=5e-4)
|
||||
finally:
|
||||
for path in (input_file, baseline_file, sp_file):
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
pass
|
||||
@@ -988,6 +988,46 @@ class TestDimensionValidation:
|
||||
assert result.shape == tensor_4d.shape
|
||||
|
||||
|
||||
@pytest.mark.cpu
|
||||
class TestStrictModeSplitValidation:
|
||||
"""Test strict mode divisibility validation in SequenceParallelSplitHook."""
|
||||
|
||||
def test_strict_mode_raises_on_non_divisible_seq_len(self):
|
||||
"""Strict Ulysses-SP should fail fast when seq_len is not divisible by sp_size."""
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig, OmniDiffusionConfig
|
||||
from vllm_omni.diffusion.distributed.sp_plan import SequenceParallelConfig
|
||||
from vllm_omni.diffusion.forward_context import set_forward_context
|
||||
from vllm_omni.diffusion.hooks.sequence_parallel import SequenceParallelSplitHook
|
||||
|
||||
class DummyModule(nn.Module):
|
||||
def forward(self, x):
|
||||
return x
|
||||
|
||||
metadata = {"x": SequenceParallelInput(split_dim=1, expected_dims=3)}
|
||||
config = SequenceParallelConfig(ulysses_degree=2, ring_degree=1)
|
||||
hook = SequenceParallelSplitHook(metadata, config)
|
||||
hook.initialize_hook(DummyModule())
|
||||
|
||||
# seq_len=5 is not divisible by sp_size=2
|
||||
x = torch.randn(1, 5, 8)
|
||||
|
||||
parallel_config = DiffusionParallelConfig(
|
||||
pipeline_parallel_size=1,
|
||||
data_parallel_size=1,
|
||||
tensor_parallel_size=1,
|
||||
sequence_parallel_size=2,
|
||||
ulysses_degree=2,
|
||||
ring_degree=1,
|
||||
cfg_parallel_size=1,
|
||||
ulysses_mode="strict",
|
||||
)
|
||||
od_config = OmniDiffusionConfig(model="test", dtype=torch.float32, parallel_config=parallel_config)
|
||||
|
||||
with set_forward_context(omni_diffusion_config=od_config):
|
||||
with pytest.raises(ValueError, match=r"strict mode.*sequence_parallel_size"):
|
||||
hook._prepare_sp_input(x, metadata["x"], (), {})
|
||||
|
||||
|
||||
@pytest.mark.cpu
|
||||
class TestSequenceParallelConfig:
|
||||
"""Test SequenceParallelConfig dataclass."""
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Perf smoke tests for Ulysses advanced_uaa communication overhead.
|
||||
|
||||
This test is intended for CI monitoring only:
|
||||
- Print per-iteration timings and ratio vs strict Ulysses all-to-all.
|
||||
- Use a loose sanity bound to catch gross regressions without flakiness.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import socket
|
||||
from dataclasses import dataclass
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
from vllm_omni.diffusion.attention.parallel.ulysses import (
|
||||
_all_gather_int,
|
||||
_ulysses_all_to_all_any_o,
|
||||
_ulysses_all_to_all_any_qkv,
|
||||
)
|
||||
from vllm_omni.diffusion.distributed.comm import SeqAllToAll4D
|
||||
from vllm_omni.diffusion.distributed.parallel_state import (
|
||||
destroy_distributed_env,
|
||||
get_sp_group,
|
||||
init_distributed_environment,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
|
||||
def _find_free_port() -> int:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
return int(s.getsockname()[1])
|
||||
|
||||
|
||||
def _set_dist_env(*, rank: int, world_size: int, master_port: int) -> None:
|
||||
os.environ["RANK"] = str(rank)
|
||||
os.environ["LOCAL_RANK"] = str(rank)
|
||||
os.environ["WORLD_SIZE"] = str(world_size)
|
||||
os.environ["MASTER_ADDR"] = "127.0.0.1"
|
||||
os.environ["MASTER_PORT"] = str(master_port)
|
||||
|
||||
|
||||
def _max_all_reduce(pg: dist.ProcessGroup, value: float, *, device: torch.device) -> float:
|
||||
t = torch.tensor([value], device=device, dtype=torch.float32)
|
||||
dist.all_reduce(t, op=dist.ReduceOp.MAX, group=pg)
|
||||
return float(t.item())
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _PerfCase:
|
||||
ulysses_degree: int
|
||||
ring_degree: int
|
||||
|
||||
@property
|
||||
def world_size(self) -> int:
|
||||
return int(self.ulysses_degree * self.ring_degree)
|
||||
|
||||
|
||||
PERF_CASES: list[_PerfCase] = [
|
||||
_PerfCase(ulysses_degree=4, ring_degree=1),
|
||||
_PerfCase(ulysses_degree=2, ring_degree=2),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", PERF_CASES)
|
||||
def test_ulysses_advanced_uaa_comm_overhead(case: _PerfCase) -> None:
|
||||
available_gpus = current_omni_platform.get_device_count()
|
||||
if available_gpus < case.world_size:
|
||||
pytest.skip(f"Requires {case.world_size} GPUs, got {available_gpus}")
|
||||
|
||||
master_port = _find_free_port()
|
||||
torch.multiprocessing.spawn(
|
||||
_perf_worker,
|
||||
args=(case.world_size, master_port, case.ulysses_degree, case.ring_degree),
|
||||
nprocs=case.world_size,
|
||||
)
|
||||
|
||||
|
||||
def _perf_worker(local_rank: int, world_size: int, master_port: int, ulysses_degree: int, ring_degree: int) -> None:
|
||||
device = torch.device(f"{current_omni_platform.device_type}:{local_rank}")
|
||||
current_omni_platform.set_device(device)
|
||||
|
||||
_set_dist_env(rank=local_rank, world_size=world_size, master_port=master_port)
|
||||
|
||||
try:
|
||||
init_distributed_environment(world_size=world_size, rank=local_rank)
|
||||
initialize_model_parallel(
|
||||
data_parallel_size=1,
|
||||
cfg_parallel_size=1,
|
||||
sequence_parallel_size=world_size,
|
||||
ulysses_degree=ulysses_degree,
|
||||
ring_degree=ring_degree,
|
||||
tensor_parallel_size=1,
|
||||
pipeline_parallel_size=1,
|
||||
)
|
||||
|
||||
sp_group = get_sp_group()
|
||||
ulysses_pg = sp_group.ulysses_group
|
||||
ring_pg = sp_group.ring_group
|
||||
|
||||
ulysses_world_size = dist.get_world_size(ulysses_pg)
|
||||
ring_world_size = dist.get_world_size(ring_pg)
|
||||
|
||||
# A moderate tensor size to reduce timing noise while staying fast in CI.
|
||||
bsz = 1
|
||||
s_local = 256
|
||||
head_cnt = 32 # divisible by ulysses_degree in both cases above
|
||||
head_dim = 128
|
||||
dtype = torch.float16
|
||||
use_sync = False
|
||||
|
||||
torch.manual_seed(1234 + local_rank)
|
||||
q = torch.randn(bsz, s_local, head_cnt, head_dim, device=device, dtype=dtype)
|
||||
k = torch.randn_like(q)
|
||||
v = torch.randn_like(q)
|
||||
|
||||
warmup_iters = 10
|
||||
iters = 200
|
||||
|
||||
def strict_comm_step(q_in: torch.Tensor, k_in: torch.Tensor, v_in: torch.Tensor) -> torch.Tensor:
|
||||
q_out = SeqAllToAll4D.apply(ulysses_pg, q_in, 2, 1, use_sync)
|
||||
SeqAllToAll4D.apply(ulysses_pg, k_in, 2, 1, use_sync)
|
||||
SeqAllToAll4D.apply(ulysses_pg, v_in, 2, 1, use_sync)
|
||||
o_in = q_out # dummy
|
||||
return SeqAllToAll4D.apply(ulysses_pg, o_in, 1, 2, use_sync)
|
||||
|
||||
def uaa_comm_step(q_in: torch.Tensor, k_in: torch.Tensor, v_in: torch.Tensor) -> torch.Tensor:
|
||||
seq_lens = _all_gather_int(ulysses_pg, int(q_in.shape[1]), device=q_in.device)
|
||||
s_global = int(sum(seq_lens))
|
||||
if ring_world_size > 1:
|
||||
ring_s_globals = _all_gather_int(ring_pg, s_global, device=q_in.device)
|
||||
if len(set(ring_s_globals)) != 1:
|
||||
raise RuntimeError(f"Unexpected hybrid ring post-Ulysses seq_len mismatch: {ring_s_globals}.")
|
||||
|
||||
q_out, orig_head_cnt = _ulysses_all_to_all_any_qkv(ulysses_pg, q_in, seq_lens=seq_lens, use_sync=use_sync)
|
||||
_ulysses_all_to_all_any_qkv(ulysses_pg, k_in, seq_lens=seq_lens, use_sync=use_sync)
|
||||
_ulysses_all_to_all_any_qkv(ulysses_pg, v_in, seq_lens=seq_lens, use_sync=use_sync)
|
||||
o_in = q_out # dummy
|
||||
return _ulysses_all_to_all_any_o(
|
||||
ulysses_pg,
|
||||
o_in,
|
||||
seq_lens=seq_lens,
|
||||
local_seq_len=int(s_local),
|
||||
orig_head_cnt=int(orig_head_cnt),
|
||||
use_sync=use_sync,
|
||||
)
|
||||
|
||||
with torch.no_grad():
|
||||
# Warmup (strict)
|
||||
for _ in range(warmup_iters):
|
||||
_ = strict_comm_step(q, k, v)
|
||||
current_omni_platform.synchronize()
|
||||
|
||||
# Timed (strict)
|
||||
t0 = torch.cuda.Event(enable_timing=True)
|
||||
t1 = torch.cuda.Event(enable_timing=True)
|
||||
t0.record()
|
||||
for _ in range(iters):
|
||||
_ = strict_comm_step(q, k, v)
|
||||
t1.record()
|
||||
current_omni_platform.synchronize()
|
||||
strict_ms = float(t0.elapsed_time(t1)) / float(iters)
|
||||
|
||||
# Warmup (UAA)
|
||||
for _ in range(warmup_iters):
|
||||
_ = uaa_comm_step(q, k, v)
|
||||
current_omni_platform.synchronize()
|
||||
|
||||
# Timed (UAA)
|
||||
u0 = torch.cuda.Event(enable_timing=True)
|
||||
u1 = torch.cuda.Event(enable_timing=True)
|
||||
u0.record()
|
||||
for _ in range(iters):
|
||||
_ = uaa_comm_step(q, k, v)
|
||||
u1.record()
|
||||
current_omni_platform.synchronize()
|
||||
uaa_ms = float(u0.elapsed_time(u1)) / float(iters)
|
||||
|
||||
# Reduce across ranks (use worst-rank to be conservative).
|
||||
strict_ms_max = _max_all_reduce(dist.group.WORLD, strict_ms, device=device)
|
||||
uaa_ms_max = _max_all_reduce(dist.group.WORLD, uaa_ms, device=device)
|
||||
ratio = (uaa_ms_max / strict_ms_max) if strict_ms_max > 0 else float("inf")
|
||||
|
||||
# Approx bytes moved per iteration per-rank (send+recv) for 4x all-to-all.
|
||||
elem_size = torch.tensor([], dtype=dtype).element_size()
|
||||
per_a2a_bytes = int(q.numel()) * elem_size * 2
|
||||
comm_bytes = int(4 * per_a2a_bytes)
|
||||
strict_gbps = (comm_bytes / (strict_ms_max / 1000.0)) / 1e9 if strict_ms_max > 0 else 0.0
|
||||
uaa_gbps = (comm_bytes / (uaa_ms_max / 1000.0)) / 1e9 if uaa_ms_max > 0 else 0.0
|
||||
|
||||
if dist.get_rank() == 0:
|
||||
payload = {
|
||||
"name": "ulysses_uaa_comm_perf",
|
||||
"world_size": int(world_size),
|
||||
"ulysses_degree": int(ulysses_degree),
|
||||
"ring_degree": int(ring_degree),
|
||||
"ulysses_world_size": int(ulysses_world_size),
|
||||
"ring_world_size": int(ring_world_size),
|
||||
"shape": {
|
||||
"bsz": int(bsz),
|
||||
"s_local": int(s_local),
|
||||
"head_cnt": int(head_cnt),
|
||||
"head_dim": int(head_dim),
|
||||
},
|
||||
"dtype": str(dtype),
|
||||
"iters": int(iters),
|
||||
"strict_ms_per_iter_max": float(strict_ms_max),
|
||||
"uaa_ms_per_iter_max": float(uaa_ms_max),
|
||||
"uaa_over_strict_ratio": float(ratio),
|
||||
"comm_bytes_per_iter_per_rank": int(comm_bytes),
|
||||
"strict_effective_gbps_per_rank": float(strict_gbps),
|
||||
"uaa_effective_gbps_per_rank": float(uaa_gbps),
|
||||
}
|
||||
print(f"UAA_COMM_PERF_JSON={payload}")
|
||||
|
||||
# Loose bound: we only want to catch severe regressions. The printed JSON
|
||||
# payload is used for monitoring smaller changes, while this threshold is
|
||||
# intentionally generous to avoid flakiness across GPU types/drivers.
|
||||
max_uaa_ms_per_iter = 10.0
|
||||
assert uaa_ms_max < max_uaa_ms_per_iter, (
|
||||
f"UAA comm too slow: uaa={uaa_ms_max:.3f}ms/iter, strict={strict_ms_max:.3f}ms/iter "
|
||||
f"(cap={max_uaa_ms_per_iter:.1f}ms/iter)."
|
||||
)
|
||||
assert ratio < 3.0, f"UAA comm overhead too high: ratio={ratio:.3f}x (strict={strict_ms_max:.3f}ms)."
|
||||
finally:
|
||||
destroy_distributed_env()
|
||||
@@ -2,8 +2,10 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import pytest
|
||||
from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
|
||||
from vllm_omni.engine.async_omni_engine import AsyncOmniEngine
|
||||
from vllm_omni.entrypoints.cli.serve import OmniServeCommand, _create_default_diffusion_stage_cfg
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
@@ -51,3 +53,43 @@ def test_default_stage_devices_from_sequence_parallel():
|
||||
)[0]
|
||||
|
||||
assert stage_cfg["runtime"]["devices"] == "0,1,2,3"
|
||||
|
||||
|
||||
def test_default_stage_config_propagates_ulysses_mode():
|
||||
"""Ensure UAA mode survives default diffusion-stage creation."""
|
||||
stage_cfg = AsyncOmniEngine._create_default_diffusion_stage_cfg(
|
||||
{
|
||||
"ulysses_degree": 4,
|
||||
"ulysses_mode": "advanced_uaa",
|
||||
}
|
||||
)[0]
|
||||
|
||||
parallel_config = stage_cfg["engine_args"]["parallel_config"]
|
||||
assert parallel_config.ulysses_degree == 4
|
||||
assert parallel_config.ulysses_mode == "advanced_uaa"
|
||||
|
||||
|
||||
def test_serve_cli_accepts_ulysses_mode():
|
||||
"""Ensure diffusion serve CLI exposes ulysses_mode and wires it to parallel_config."""
|
||||
parser = FlexibleArgumentParser()
|
||||
subparsers = parser.add_subparsers(dest="command")
|
||||
OmniServeCommand().subparser_init(subparsers)
|
||||
|
||||
args = parser.parse_args(
|
||||
[
|
||||
"serve",
|
||||
"Qwen/Qwen-Image",
|
||||
"--omni",
|
||||
"--usp",
|
||||
"4",
|
||||
"--ulysses-mode",
|
||||
"advanced_uaa",
|
||||
]
|
||||
)
|
||||
|
||||
stage_cfg = _create_default_diffusion_stage_cfg(args)[0]
|
||||
parallel_config = stage_cfg["engine_args"]["parallel_config"]
|
||||
|
||||
assert args.ulysses_mode == "advanced_uaa"
|
||||
assert parallel_config.ulysses_degree == 4
|
||||
assert parallel_config.ulysses_mode == "advanced_uaa"
|
||||
@@ -62,6 +62,21 @@ def pytorch_attn_forward(
|
||||
if op_type == "flash" and q.dtype == torch.float32:
|
||||
op_type = "efficient"
|
||||
|
||||
# Volta (V100, sm70) does not have native bf16 support for the efficient
|
||||
# SDPA kernel. When running Ring attention in bf16 on such GPUs, the
|
||||
# efficient kernel can fail with "cutlassF: no kernel found to launch".
|
||||
#
|
||||
# We keep the overall model in bf16 for numerical stability, but cast
|
||||
# the attention inputs to fp16 for the SDPA call. The output is cast
|
||||
# back to bf16 to match the caller's expected dtype.
|
||||
orig_dtype = q.dtype
|
||||
if q.is_cuda and q.dtype == torch.bfloat16 and op_type == "efficient":
|
||||
major_cc, _minor_cc = torch.cuda.get_device_capability(q.device)
|
||||
if major_cc < 8:
|
||||
q = q.to(torch.float16)
|
||||
k = k.to(torch.float16)
|
||||
v = v.to(torch.float16)
|
||||
|
||||
q = q.transpose(1, 2)
|
||||
k = k.transpose(1, 2)
|
||||
v = v.transpose(1, 2)
|
||||
@@ -90,7 +105,13 @@ def pytorch_attn_forward(
|
||||
raise ValueError(f"Invalid op_type: {op_type}")
|
||||
|
||||
out = out.transpose(1, 2)
|
||||
lse = lse.to(q.dtype)
|
||||
# Keep LSE in fp32 for numerical stability when accumulating across ring
|
||||
# steps (update_out_and_lse uses sigmoid/logsigmoid on LSE diffs).
|
||||
# Casting LSE down to fp16/bf16 can introduce NaNs on some GPUs/shapes.
|
||||
lse = lse.to(torch.float32)
|
||||
|
||||
if out.dtype != orig_dtype:
|
||||
out = out.to(orig_dtype)
|
||||
|
||||
return out, lse
|
||||
|
||||
|
||||
@@ -7,11 +7,143 @@ from dataclasses import dataclass
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
import torch.nn.functional as F
|
||||
|
||||
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
|
||||
from vllm_omni.diffusion.attention.parallel.base import ParallelAttentionContext
|
||||
from vllm_omni.diffusion.distributed.comm import SeqAllToAll4D
|
||||
from vllm_omni.diffusion.distributed.group_coordinator import SequenceParallelGroupCoordinator
|
||||
from vllm_omni.diffusion.forward_context import get_ulysses_mode
|
||||
|
||||
|
||||
def _ceil_div(n: int, d: int) -> int:
|
||||
return (n + d - 1) // d
|
||||
|
||||
|
||||
def _positive_divisors(n: int) -> list[int]:
|
||||
if n <= 0:
|
||||
return []
|
||||
divs = set()
|
||||
i = 1
|
||||
while i * i <= n:
|
||||
if n % i == 0:
|
||||
divs.add(i)
|
||||
divs.add(n // i)
|
||||
i += 1
|
||||
return sorted(divs)
|
||||
|
||||
|
||||
@torch.compiler.disable
|
||||
def _all_gather_int(pg: dist.ProcessGroup, value: int, *, device: torch.device) -> list[int]:
|
||||
"""All-gather a scalar int across pg.
|
||||
|
||||
Note: we use a device tensor so this works for NCCL subgroups (e.g. Ulysses/Ring).
|
||||
"""
|
||||
world_size = dist.get_world_size(pg)
|
||||
if world_size == 1:
|
||||
return [int(value)]
|
||||
|
||||
t = torch.tensor([int(value)], dtype=torch.int64, device=device)
|
||||
gathered = [torch.empty_like(t) for _ in range(world_size)]
|
||||
dist.all_gather(gathered, t, group=pg)
|
||||
return [int(x.item()) for x in gathered]
|
||||
|
||||
|
||||
def _ulysses_all_to_all_any_qkv(
|
||||
pg: dist.ProcessGroup,
|
||||
x: torch.Tensor, # (B, S_local, H, D)
|
||||
*,
|
||||
seq_lens: list[int],
|
||||
use_sync: bool,
|
||||
) -> tuple[torch.Tensor, int]:
|
||||
"""UAA forward all-to-all: (B, S_local, H, D) -> (B, S_global, H_local, D).
|
||||
|
||||
Returns:
|
||||
(resharded, orig_head_cnt)
|
||||
"""
|
||||
world_size = dist.get_world_size(pg)
|
||||
if world_size == 1:
|
||||
return x, int(x.shape[2])
|
||||
|
||||
bsz, s_local, head_cnt, head_dim = x.shape
|
||||
orig_head_cnt = int(head_cnt)
|
||||
padded_head_cnt = _ceil_div(orig_head_cnt, world_size) * world_size
|
||||
head_pad = padded_head_cnt - orig_head_cnt
|
||||
if head_pad:
|
||||
x = F.pad(x, (0, 0, 0, head_pad))
|
||||
|
||||
head_cnt_local = padded_head_cnt // world_size
|
||||
|
||||
# (B, S_local, H, D) -> (world_size, S_local, B, H_local, D)
|
||||
x_t = x.reshape(bsz, s_local, world_size, head_cnt_local, head_dim).permute(2, 1, 0, 3, 4).contiguous()
|
||||
# (world_size, S_local, B, H_local, D) -> (world_size * S_local, B, H_local, D)
|
||||
x_t = x_t.flatten(0, 1)
|
||||
|
||||
input_split_sizes = [s_local] * world_size
|
||||
output_split_sizes = seq_lens
|
||||
s_global = int(sum(output_split_sizes))
|
||||
|
||||
out = torch.empty((s_global, bsz, head_cnt_local, head_dim), device=x.device, dtype=x.dtype)
|
||||
dist.all_to_all_single(
|
||||
out,
|
||||
x_t,
|
||||
output_split_sizes=output_split_sizes,
|
||||
input_split_sizes=input_split_sizes,
|
||||
group=pg,
|
||||
)
|
||||
if use_sync:
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
current_omni_platform.synchronize()
|
||||
|
||||
# (S_global, B, H_local, D) -> (B, S_global, H_local, D)
|
||||
out = out.permute(1, 0, 2, 3).contiguous()
|
||||
return out, orig_head_cnt
|
||||
|
||||
|
||||
def _ulysses_all_to_all_any_o(
|
||||
pg: dist.ProcessGroup,
|
||||
x: torch.Tensor, # (B, S_global, H_local, D)
|
||||
*,
|
||||
seq_lens: list[int],
|
||||
local_seq_len: int,
|
||||
orig_head_cnt: int,
|
||||
use_sync: bool,
|
||||
) -> torch.Tensor:
|
||||
"""UAA reverse all-to-all: (B, S_global, H_local, D) -> (B, S_local, H, D)."""
|
||||
world_size = dist.get_world_size(pg)
|
||||
if world_size == 1:
|
||||
return x
|
||||
|
||||
bsz, s_global, head_cnt_local, head_dim = x.shape
|
||||
s_local = int(local_seq_len)
|
||||
|
||||
# (B, S_global, H_local, D) -> (S_global, B, H_local, D)
|
||||
x_t = x.permute(1, 0, 2, 3).contiguous()
|
||||
|
||||
input_split_sizes = seq_lens
|
||||
output_split_sizes = [s_local] * world_size
|
||||
|
||||
out = torch.empty((world_size * s_local, bsz, head_cnt_local, head_dim), device=x.device, dtype=x.dtype)
|
||||
dist.all_to_all_single(
|
||||
out,
|
||||
x_t,
|
||||
output_split_sizes=output_split_sizes,
|
||||
input_split_sizes=input_split_sizes,
|
||||
group=pg,
|
||||
)
|
||||
if use_sync:
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
current_omni_platform.synchronize()
|
||||
|
||||
# (world_size * S_local, B, H_local, D) -> (B, S_local, H, D)
|
||||
out = out.reshape(world_size, s_local, bsz, head_cnt_local, head_dim).permute(2, 1, 0, 3, 4).contiguous()
|
||||
out = out.reshape(bsz, s_local, world_size * head_cnt_local, head_dim)
|
||||
|
||||
if out.shape[2] != orig_head_cnt:
|
||||
out = out[:, :, :orig_head_cnt, :].contiguous()
|
||||
return out
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -24,6 +156,12 @@ class _UlyssesCtx(ParallelAttentionContext):
|
||||
use_sync: bool
|
||||
joint_len: int = 0
|
||||
joint_strategy: str = "front"
|
||||
# UAA (Ulysses Anything Attention) metadata
|
||||
use_uaa: bool = False
|
||||
uaa_seq_lens: tuple[int, ...] = ()
|
||||
uaa_local_seq_len: int = 0
|
||||
orig_head_cnt: int = 0
|
||||
joint_orig_head_cnt: int = 0
|
||||
|
||||
|
||||
class UlyssesParallelAttention:
|
||||
@@ -65,9 +203,11 @@ class UlyssesParallelAttention:
|
||||
value: torch.Tensor,
|
||||
attn_metadata: AttentionMetadata | None,
|
||||
):
|
||||
mode = get_ulysses_mode(default="strict")
|
||||
joint_tensor_query = joint_tensor_key = joint_tensor_value = None
|
||||
joint_strategy = "front"
|
||||
joint_len = 0
|
||||
joint_orig_head_cnt = 0
|
||||
|
||||
if attn_metadata is not None:
|
||||
joint_tensor_query = attn_metadata.joint_query
|
||||
@@ -88,7 +228,27 @@ class UlyssesParallelAttention:
|
||||
# joint_query is (B, S, H, D). We split H (dim 2).
|
||||
ulysses_world_size = self._sp_group.ulysses_world_size
|
||||
ulysses_rank = self._sp_group.ulysses_rank
|
||||
attn_heads_per_ulysses_rank = joint_tensor_query.shape[-2] // ulysses_world_size
|
||||
joint_head_cnt = int(joint_tensor_query.shape[-2])
|
||||
joint_orig_head_cnt = joint_head_cnt
|
||||
|
||||
if mode == "advanced_uaa":
|
||||
padded_joint_head_cnt = _ceil_div(joint_head_cnt, ulysses_world_size) * ulysses_world_size
|
||||
joint_head_pad = padded_joint_head_cnt - joint_head_cnt
|
||||
if joint_head_pad:
|
||||
joint_tensor_query = F.pad(joint_tensor_query, (0, 0, 0, joint_head_pad))
|
||||
joint_tensor_key = F.pad(joint_tensor_key, (0, 0, 0, joint_head_pad))
|
||||
joint_tensor_value = F.pad(joint_tensor_value, (0, 0, 0, joint_head_pad))
|
||||
joint_head_cnt = padded_joint_head_cnt
|
||||
else:
|
||||
if joint_head_cnt % ulysses_world_size != 0:
|
||||
supported = _positive_divisors(joint_head_cnt)
|
||||
raise ValueError(
|
||||
"Ulysses-SP strict mode requires joint head_cnt divisible by ulysses_degree. "
|
||||
f"joint_head_cnt={joint_head_cnt}, ulysses_degree={ulysses_world_size}. "
|
||||
f"Try ulysses_degree in {supported}, or set ulysses_mode='advanced_uaa'."
|
||||
)
|
||||
|
||||
attn_heads_per_ulysses_rank = joint_head_cnt // ulysses_world_size
|
||||
|
||||
# Note: We use the same heads for Q/K/V
|
||||
joint_tensor_query = joint_tensor_query[
|
||||
@@ -126,10 +286,55 @@ class UlyssesParallelAttention:
|
||||
attn_metadata.joint_key = joint_tensor_key
|
||||
attn_metadata.joint_value = joint_tensor_value
|
||||
|
||||
# (bs, seq_len/P, head_cnt, head_size) -> (bs, seq_len, head_cnt/P, head_size)
|
||||
query = SeqAllToAll4D.apply(self._ulysses_pg, query, self._scatter_idx, self._gather_idx, self._use_sync)
|
||||
key = SeqAllToAll4D.apply(self._ulysses_pg, key, self._scatter_idx, self._gather_idx, self._use_sync)
|
||||
value = SeqAllToAll4D.apply(self._ulysses_pg, value, self._scatter_idx, self._gather_idx, self._use_sync)
|
||||
ulysses_world_size = self._sp_group.ulysses_world_size
|
||||
if mode == "advanced_uaa":
|
||||
if self._scatter_idx != 2 or self._gather_idx != 1:
|
||||
raise ValueError(
|
||||
"ulysses_mode='advanced_uaa' currently only supports scatter_idx=2, gather_idx=1 "
|
||||
f"(got scatter_idx={self._scatter_idx}, gather_idx={self._gather_idx})."
|
||||
)
|
||||
|
||||
local_seq_len = int(query.shape[1])
|
||||
seq_lens = _all_gather_int(self._ulysses_pg, local_seq_len, device=query.device)
|
||||
s_global = int(sum(seq_lens))
|
||||
|
||||
# In hybrid Ulysses+Ring, Ring attention uses P2P send/recv with fixed-shape
|
||||
# buffers. This requires all ring ranks to have the same seq_len after the
|
||||
# Ulysses all-to-all (i.e. per-ring-rank S_global must match).
|
||||
if self._sp_group.ring_world_size > 1:
|
||||
ring_s_globals = _all_gather_int(self._sp_group.ring_group, s_global, device=query.device)
|
||||
if len(set(ring_s_globals)) != 1:
|
||||
raise ValueError(
|
||||
"ulysses_mode='advanced_uaa' with hybrid Ulysses+Ring requires the "
|
||||
"post-Ulysses seq_len to be equal across ring ranks, but got "
|
||||
f"{ring_s_globals} (ring_degree={self._sp_group.ring_world_size}). "
|
||||
"This typically means the input sequence was not evenly shardable across the ring. "
|
||||
"Try setting ring_degree=1, or choose a sequence length divisible by ring_degree."
|
||||
)
|
||||
query, orig_head_cnt = _ulysses_all_to_all_any_qkv(
|
||||
self._ulysses_pg, query, seq_lens=seq_lens, use_sync=self._use_sync
|
||||
)
|
||||
key, _ = _ulysses_all_to_all_any_qkv(self._ulysses_pg, key, seq_lens=seq_lens, use_sync=self._use_sync)
|
||||
value, _ = _ulysses_all_to_all_any_qkv(self._ulysses_pg, value, seq_lens=seq_lens, use_sync=self._use_sync)
|
||||
else:
|
||||
# Strict mode: fail fast with actionable errors for head divisibility.
|
||||
for name, t in (("query", query), ("key", key), ("value", value)):
|
||||
head_cnt = int(t.shape[2])
|
||||
if head_cnt % ulysses_world_size != 0:
|
||||
supported = _positive_divisors(head_cnt)
|
||||
raise ValueError(
|
||||
"Ulysses-SP strict mode requires head_cnt divisible by ulysses_degree. "
|
||||
f"{name}_head_cnt={head_cnt}, ulysses_degree={ulysses_world_size}. "
|
||||
f"Try ulysses_degree in {supported}, or set ulysses_mode='advanced_uaa'."
|
||||
)
|
||||
|
||||
# (bs, seq_len/P, head_cnt, head_size) -> (bs, seq_len, head_cnt/P, head_size)
|
||||
query = SeqAllToAll4D.apply(self._ulysses_pg, query, self._scatter_idx, self._gather_idx, self._use_sync)
|
||||
key = SeqAllToAll4D.apply(self._ulysses_pg, key, self._scatter_idx, self._gather_idx, self._use_sync)
|
||||
value = SeqAllToAll4D.apply(self._ulysses_pg, value, self._scatter_idx, self._gather_idx, self._use_sync)
|
||||
seq_lens = []
|
||||
local_seq_len = 0
|
||||
orig_head_cnt = 0
|
||||
|
||||
if is_joint:
|
||||
# Concatenate joint query AFTER AllToAll
|
||||
@@ -162,6 +367,11 @@ class UlyssesParallelAttention:
|
||||
use_sync=self._use_sync,
|
||||
joint_len=joint_len,
|
||||
joint_strategy=joint_strategy,
|
||||
use_uaa=(mode == "advanced_uaa"),
|
||||
uaa_seq_lens=tuple(int(x) for x in seq_lens) if mode == "advanced_uaa" else (),
|
||||
uaa_local_seq_len=int(local_seq_len) if mode == "advanced_uaa" else 0,
|
||||
orig_head_cnt=int(orig_head_cnt) if mode == "advanced_uaa" else 0,
|
||||
joint_orig_head_cnt=int(joint_orig_head_cnt) if mode == "advanced_uaa" else 0,
|
||||
)
|
||||
|
||||
if attn_metadata is not None:
|
||||
@@ -217,7 +427,19 @@ class UlyssesParallelAttention:
|
||||
# SeqAllToAll4D handles: Scatter gather_idx, Gather scatter_idx.
|
||||
# Forward: Scatter 2 (H), Gather 1 (S).
|
||||
# Reverse: Scatter 1 (S), Gather 2 (H).
|
||||
output_img = SeqAllToAll4D.apply(ctx.ulysses_pg, output_img, ctx.gather_idx, ctx.scatter_idx, ctx.use_sync)
|
||||
if ctx.use_uaa:
|
||||
output_img = _ulysses_all_to_all_any_o(
|
||||
ctx.ulysses_pg,
|
||||
output_img,
|
||||
seq_lens=list(ctx.uaa_seq_lens),
|
||||
local_seq_len=ctx.uaa_local_seq_len,
|
||||
orig_head_cnt=ctx.orig_head_cnt,
|
||||
use_sync=ctx.use_sync,
|
||||
)
|
||||
else:
|
||||
output_img = SeqAllToAll4D.apply(
|
||||
ctx.ulysses_pg, output_img, ctx.gather_idx, ctx.scatter_idx, ctx.use_sync
|
||||
)
|
||||
|
||||
# 2. Process Joint part: AllGather on Heads
|
||||
# Input: (B, JointLen, H/P, D). Output: (B, JointLen, H, D).
|
||||
@@ -227,6 +449,8 @@ class UlyssesParallelAttention:
|
||||
gathered_joint = [torch.zeros_like(output_joint) for _ in range(dist.get_world_size(ctx.ulysses_pg))]
|
||||
dist.all_gather(gathered_joint, output_joint, group=ctx.ulysses_pg)
|
||||
output_joint = torch.cat(gathered_joint, dim=2)
|
||||
if ctx.use_uaa and ctx.joint_orig_head_cnt > 0 and output_joint.shape[2] != ctx.joint_orig_head_cnt:
|
||||
output_joint = output_joint[:, :, : ctx.joint_orig_head_cnt, :].contiguous()
|
||||
|
||||
# 3. Recombine
|
||||
if ctx.joint_strategy == "front":
|
||||
@@ -235,4 +459,13 @@ class UlyssesParallelAttention:
|
||||
return torch.cat([output_img, output_joint], dim=1)
|
||||
|
||||
# Standard Ulysses Reverse
|
||||
if ctx.use_uaa:
|
||||
return _ulysses_all_to_all_any_o(
|
||||
ctx.ulysses_pg,
|
||||
attn_output,
|
||||
seq_lens=list(ctx.uaa_seq_lens),
|
||||
local_seq_len=ctx.uaa_local_seq_len,
|
||||
orig_head_cnt=ctx.orig_head_cnt,
|
||||
use_sync=ctx.use_sync,
|
||||
)
|
||||
return SeqAllToAll4D.apply(ctx.ulysses_pg, attn_output, ctx.gather_idx, ctx.scatter_idx, ctx.use_sync)
|
||||
@@ -55,6 +55,20 @@ class DiffusionParallelConfig:
|
||||
ring_degree: int = 1
|
||||
"""Number of GPUs used for ring sequence parallelism."""
|
||||
|
||||
ulysses_mode: str = "strict"
|
||||
"""Ulysses sequence-parallel mode.
|
||||
|
||||
- "strict": Require divisibility constraints (fastest, default).
|
||||
- "advanced_uaa": Enable UAA ("Ulysses Anything Attention") to support
|
||||
uneven sequence lengths and non-divisible head counts.
|
||||
|
||||
Note:
|
||||
- Ring attention does not support `attention_mask`, so models that rely on
|
||||
mask-based auto-padding are still incompatible with Ring.
|
||||
- When used in hybrid Ulysses+Ring, Ring requires consistent per-rank
|
||||
sequence shapes across the ring group.
|
||||
"""
|
||||
|
||||
cfg_parallel_size: int = 1
|
||||
"""Number of Classifier Free Guidance (CFG) parallel groups."""
|
||||
|
||||
@@ -88,6 +102,9 @@ class DiffusionParallelConfig:
|
||||
"Sequence parallel size must be equal to the product of ulysses degree and ring degree,"
|
||||
f" but got {self.sequence_parallel_size} != {self.ulysses_degree} * {self.ring_degree}"
|
||||
)
|
||||
assert self.ulysses_mode in {"strict", "advanced_uaa"}, (
|
||||
f"ulysses_mode must be one of {{'strict','advanced_uaa'}}, but got {self.ulysses_mode!r}."
|
||||
)
|
||||
|
||||
# Validate HSDP configuration
|
||||
if self.use_hsdp:
|
||||
|
||||
@@ -80,6 +80,23 @@ def is_forward_context_available() -> bool:
|
||||
return _forward_context is not None
|
||||
|
||||
|
||||
def get_ulysses_mode(*, default: str = "strict") -> str:
|
||||
"""Resolve the Ulysses-SP mode from the current ForwardContext.
|
||||
|
||||
Returns `default` when ForwardContext is unavailable or the diffusion
|
||||
config is not set.
|
||||
"""
|
||||
if not is_forward_context_available():
|
||||
return default
|
||||
|
||||
cfg = get_forward_context().omni_diffusion_config
|
||||
if cfg is None:
|
||||
return default
|
||||
|
||||
parallel_config = cfg.parallel_config
|
||||
return str(getattr(parallel_config, "ulysses_mode", default))
|
||||
|
||||
|
||||
def create_forward_context(
|
||||
vllm_config: VllmConfig | None = None,
|
||||
omni_diffusion_config: OmniDiffusionConfig | None = None,
|
||||
|
||||
@@ -315,10 +315,38 @@ class SequenceParallelSplitHook(ModelHook):
|
||||
logger.warning_once(f"Expected tensor with {sp_input.expected_dims} dims, got {x.dim()}. Skipping split.")
|
||||
return x
|
||||
|
||||
def _raise_strict_divisibility_error(*, dim: int, seq_len: int, sp_size: int) -> None:
|
||||
# Keep message actionable: strict mode must be evenly shardable at the split hook level.
|
||||
msg = (
|
||||
"Sequence Parallel strict mode requires the sequence length to be evenly shardable at split time, "
|
||||
f"but got seq_len={seq_len} (dim={dim}) not divisible by sequence_parallel_size={sp_size}. "
|
||||
)
|
||||
suggestions: list[str] = []
|
||||
if self.config.ulysses_degree > 1:
|
||||
suggestions.append("set parallel_config.ulysses_mode='advanced_uaa' (UAA)")
|
||||
suggestions.append(
|
||||
"enable auto_pad=True in the model _sp_plan (requires attention_mask support and ring_degree=1)"
|
||||
)
|
||||
suggestions.append("choose seq_len divisible by sequence_parallel_size (or adjust SP degrees)")
|
||||
msg += "Try: " + "; ".join(suggestions) + "."
|
||||
raise ValueError(msg)
|
||||
|
||||
def _maybe_validate_strict_divisibility(*, dim: int, seq_len: int) -> None:
|
||||
from vllm_omni.diffusion.forward_context import get_ulysses_mode
|
||||
|
||||
if get_ulysses_mode(default="strict") != "strict":
|
||||
return
|
||||
sp_size = int(self.config.sequence_parallel_size)
|
||||
if sp_size <= 1:
|
||||
return
|
||||
if seq_len < sp_size or (seq_len % sp_size) != 0:
|
||||
_raise_strict_divisibility_error(dim=dim, seq_len=int(seq_len), sp_size=sp_size)
|
||||
|
||||
if isinstance(sp_input, SequenceParallelInput):
|
||||
# Full split with optional auto-padding
|
||||
if sp_input.auto_pad:
|
||||
return self._shard_with_auto_pad(x, sp_input.split_dim)
|
||||
_maybe_validate_strict_divisibility(dim=sp_input.split_dim, seq_len=x.size(sp_input.split_dim))
|
||||
return sp_shard(x, sp_input.split_dim, validate=False)
|
||||
elif isinstance(sp_input, SequenceParallelPartialInput):
|
||||
# Partial split: keep text portion, split image portion
|
||||
@@ -330,6 +358,7 @@ class SequenceParallelSplitHook(ModelHook):
|
||||
image_part = x.narrow(dim, text_len, x.size(dim) - text_len)
|
||||
|
||||
# Only shard the image portion
|
||||
_maybe_validate_strict_divisibility(dim=dim, seq_len=image_part.size(dim))
|
||||
image_part_sharded = sp_shard(image_part, dim, validate=False)
|
||||
|
||||
# Concatenate back: [text_full, image_sharded]
|
||||
|
||||
@@ -623,6 +623,11 @@ class ZImageTransformer2DModel(CachedTransformer):
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
# NOTE: `DiffusersPipelineLoader.load_model()` initializes this module
|
||||
# under `set_default_torch_dtype(od_config.dtype)`, so using the current
|
||||
# default dtype keeps `self.dtype` consistent with the actually loaded
|
||||
# weights. This dtype is used by the pipeline to cast inputs.
|
||||
self.dtype = torch.get_default_dtype()
|
||||
self.in_channels = in_channels
|
||||
self.out_channels = in_channels
|
||||
self.all_patch_size = all_patch_size
|
||||
|
||||
@@ -802,6 +802,7 @@ class AsyncOmniEngine:
|
||||
if parallel_config is None:
|
||||
ulysses_degree = normalized_kwargs.get("ulysses_degree") or 1
|
||||
ring_degree = normalized_kwargs.get("ring_degree") or 1
|
||||
ulysses_mode = normalized_kwargs.get("ulysses_mode") or "strict"
|
||||
sequence_parallel_size = normalized_kwargs.get("sequence_parallel_size")
|
||||
tensor_parallel_size = normalized_kwargs.get("tensor_parallel_size") or 1
|
||||
cfg_parallel_size = normalized_kwargs.get("cfg_parallel_size") or 1
|
||||
@@ -819,6 +820,7 @@ class AsyncOmniEngine:
|
||||
sequence_parallel_size=sequence_parallel_size,
|
||||
ulysses_degree=ulysses_degree,
|
||||
ring_degree=ring_degree,
|
||||
ulysses_mode=ulysses_mode,
|
||||
cfg_parallel_size=cfg_parallel_size,
|
||||
vae_patch_parallel_size=vae_patch_parallel_size,
|
||||
use_hsdp=use_hsdp,
|
||||
|
||||
@@ -200,6 +200,15 @@ class OmniServeCommand(CLISubcommand):
|
||||
help="Ulysses Sequence Parallelism degree for diffusion models. "
|
||||
"Equivalent to setting DiffusionParallelConfig.ulysses_degree.",
|
||||
)
|
||||
omni_config_group.add_argument(
|
||||
"--ulysses-mode",
|
||||
type=str,
|
||||
default="strict",
|
||||
choices=["strict", "advanced_uaa"],
|
||||
help="Ulysses sequence-parallel mode for diffusion models. "
|
||||
"'strict' keeps the original divisibility requirements; "
|
||||
"'advanced_uaa' enables the experimental UAA path for uneven sequence/head shapes.",
|
||||
)
|
||||
omni_config_group.add_argument(
|
||||
"--ring",
|
||||
"--ring-degree",
|
||||
|
||||
Reference in new issue
Block a user