[Model] Add LingBot World Ulysses sequence parallelism (#6841)

Signed-off-by: wtz2333 <2955110911@qq.com>
Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk>
This commit is contained in:
wtz2333andZhou Taichang authored and GitHub committed 2026-09-11 18:37:12 +08:00
1 parent de9a133d48
commit 02aaa34059
8 files changed
+659 -45

No files matched your search

@@ -178,6 +178,24 @@ For LingBot World v2:
- `lingbot_world/pipeline.py` constructs conditioning, owns small non-KV session state, and
produces one block plus the standard metadata envelope.
## Ulysses sequence parallelism
LingBot supports pure Ulysses sequence parallelism for both direct and
AR-Diffusion execution. With Ulysses degree greater than one, hidden tokens,
camera features, token-expanded timestep modulation, and RoPE tables are sharded
together. Without SP, timestep modulation retains the frame-broadcast path.
Self-attention performs the sequence-to-head all-to-all before reading or writing
paged KV, while static text K/V uses the same local head shard. Text K/V shards
own compact storage; cross-attention exchanges query/output layouts to use these
shards. This retains the shared cache geometry and reduces text K/V storage at
the cost of two all-to-all calls per layer.
The output head projects local tokens before gathering the flow values. For
the 14B model this reduces the gathered width from 5120 to 64; frame modulation
uses each shard's global token offset, including shards that split a frame. Only
`ulysses_mode="strict"` is supported; `advanced_uaa`, Ring, and AllGather-KV modes
remain unsupported for this model.
## Non-goals
This contract does not currently provide: