Compare commits

...
14 changed files with 353 additions and 63 deletions

No files matched your search

+1
View File
@@ -120,6 +120,7 @@ nav:
- design/feature/omni_async_output_materialization.md
- design/feature/vae_parallel.md
- design/feature/diffusion_continuous_batching.md
- design/feature/distributed_layerwise_offload.md
- Module Design:
- design/module/ar_module.md
- design/module/dit_module.md
@@ -0,0 +1,156 @@
# Distributed Layerwise Offload
This document describes distributed layerwise offload (DLO) for diffusion
models. DLO keeps only a small number of DiT blocks on the accelerator and
streams the remaining blocks from host memory. The distributed backend can
either shard those host-side weights across an existing parallel group or keep
the standard loader's rank-local weights and avoid an additional collective.
For user-facing commands, see the
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload_diffusion.md)
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
## Status
DLO is implemented for multi-device diffusion execution. The default
AllGather path is the primary path for DP and SP deployments. The
`--dlo-no-use-allgather` path is a rank-local compatibility mode: it is useful
for standard-loader sharding, workstation bring-up, and systems where an
additional DLO collective is undesirable, but it does not reduce host weight
storage across ranks.
The compatibility matrix below describes the current implementation. The
unit-level guards are covered, but not every parallelism combination has a
full model-and-hardware end-to-end test.
## Design
### DLO consumes the existing parallel topology
DLO does not create a new DP, TP, or SP topology. It reads the configured
`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
standard distributed groups have been initialized.
The DLO weight-sharding group is selected as follows:
1. Use the existing DP group when `data_parallel_size > 1`.
2. When DP is one and SP is greater than one, use the SP group.
3. Otherwise, run rank-locally without a DLO process group.
TP is deliberately not used as DLO's AllGather group. HSDP has its own
parameter-sharding lifecycle and is not allowed to be sharded a second time by
DLO's AllGather path.
### AllGather path
With the default `dlo_use_allgather=True`, each rank stores approximately
`1 / group_size` of each streamable block in pinned host memory. The next
block's shard is copied to a device buffer and reconstructed with
`all_gather_into_tensor` on a communication stream while the current block is
executing.
```text
Compute: [Block N] [Block N+1] [Block N+2]
H2D: [shard N+1] [shard N+2]
AllGather: [full N+1] [full N+2]
Buffers: [current slot] [prefetch slot] [current slot]
```
![DLO double-buffer prefetch pipeline](../figures/dlo/dlo_pipeline.gif)
The backend uses two shared device buffers, so accelerator weight residency is
bounded by the largest streamed blocks rather than the complete model.
When DP is greater than one, the engine can process one request per DP rank in
the same denoising wave. Because AllGather is a collective, all participating
requests must take the same execution path at every denoising step.
### Rank-local path without DLO AllGather
With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
one. The regular model loader remains responsible for preparing each rank's
weights, including TP-local tensors or HSDP-managed parameters. DLO then
streams those rank-local tensors block by block using H2D copies only.
This mode means:
- DP still provides independent replicas, but DLO does not shard weights
across DP ranks.
- SP still performs its normal activation/attention collectives, but DLO does
not shard weights across SP ranks.
- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
DLO's additional weight AllGather is disabled.
- Pure DP deployments keep a full host-side model copy per rank, subject to
shared OS page-cache behavior.
- The scheduler does not require a synchronized DP request wave for DLO.
This path is intentionally not implemented by reusing the AllGather mmap
loader. It relies on the standard loader so model-specific TP/HSDP transforms
remain intact.
## Parallelism compatibility
| Parallelism | DLO + AllGather | DLO without AllGather |
|---|---|---|
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. DP replicas remain independent; no DLO weight sharding or cross-DP DLO collective. |
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
| **TP > 1** | Unsupported in the DLO mmap path. TP-aware loader callbacks are bypassed, so the backend rejects this configuration when it enters that path. | Standard loading is retained and TP-local tensors can be streamed. This is the intended compatibility path, but it still needs broader model and hardware validation. |
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
### Combined dimensions
- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
than one; SP continues to use its own sequence-parallel group. If DP is one,
the SP group becomes DLO's sharding group in AllGather mode.
- **DP + TP/SP without AllGather:** standard model loading defines the
rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
collective.
- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
materialization and synchronization.
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
configuration.
## Request and loading constraints
AllGather DP multi-concurrency requires:
- explicit `num_inference_steps`;
- the same `num_inference_steps` for all requests in a wave; and
- identical request arguments that affect the collective execution path.
The no-AllGather path does not impose these DLO-specific synchronized-wave
requirements.
The mmap loader is used only by the supported DLO+AllGather path when the
model has a compatible checkpoint layout. Online quantization is incompatible
with that sharded mmap path; use `--dlo-no-use-allgather` or disable online
quantization. Models that do not meet the mmap requirements use the regular
loader path.
## Validation coverage
Current source-level validation includes:
- HSDP + DLO + AllGather rejection;
- HSDP + DLO without AllGather acceptance at configuration level;
- TP rejection in the DLO+AllGather mmap path;
- resident-layer requests requiring no-AllGather;
- DP request-wave validation for denoising-step compatibility;
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
tests.
The highest-value missing coverage is end-to-end numerical comparison against
ordinary layerwise offload for DP+SP, TP+no-AllGather, and HSDP+SP+no-AllGather
on the target CUDA/NCCL or CANN/HCCL hardware.
## Recommendations
- Use **DP + DLO AllGather** for the supported throughput and host-memory
scaling path.
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
not the goal.
- Use **no-AllGather** to bring up TP or workstation PCIe configurations, with
the expectation of higher host-memory use and lower validation confidence.
- Prefer **HSDP alone** for production HSDP deployments until the combined
HSDP + DLO no-AllGather path has broader end-to-end coverage.
Binary file not shown.

After

Width:  |  Height:  |  Size: 195 KiB

+1
View File
@@ -12,6 +12,7 @@ This section contains design documents and architecture specifications for vLLM-
- [Ray-based Execution](feature/ray_based_execution.md)
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
## Infrastructure Design Documents
@@ -184,10 +184,15 @@ class Flux2Transformer2DModel(nn.Module):
### How It Works
Distributed layerwise offloading extends single-GPU layerwise offloading to
multi-device deployments. Each DP rank stores only **1/dp_size** of the model
weights in host memory; full layer weights are reconstructed at runtime via
**AllGather** on a dedicated communication stream, overlapped with computation
via a fixed double-buffer scheme.
multi-device deployments. In the default mode, each rank stores only
**1/dp_size** of the model weights in host memory; full layer weights are
reconstructed at runtime via **AllGather** on a dedicated communication stream,
overlapped with computation via a fixed double-buffer scheme. The
`--dlo-no-use-allgather` mode instead streams standard-loader rank-local
weights independently and does not shard weights across ranks.
For the implementation design and compatibility matrix, see
[Distributed Layerwise Offload](../../design/feature/distributed_layerwise_offload.md).
**Key features:**
- **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
@@ -219,7 +224,7 @@ vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4
# Without AllGather (each rank loads full weights, no sharding)
# Without DLO AllGather (each rank streams standard-loader rank-local weights)
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4 \
@@ -247,9 +252,9 @@ m = Omni(
| Flag | Description | Default |
|------|-------------|---------|
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
| `--data-parallel-size N` | Number of DP ranks for weight sharding | `1` |
| `--data-parallel-size N` | Number of DP ranks; DLO weight-sharding group when AllGather is enabled | `1` |
| `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
| `--dlo-no-use-allgather` | Full weights per rank (no sharding, no AllGather) | `false` |
| `--dlo-no-use-allgather` | Use standard-loader rank-local weights; no DLO sharding or AllGather | `false` |
### How model weights are loaded (mmap path)
@@ -303,9 +308,15 @@ request while AllGather synchronizes only weight shards (request-independent).
### Limitations
- Online quantization (FP8) is incompatible with mmap loading — falls back to
regular `load_weights()` automatically
- Tensor Parallel is not supported (DLO uses DP-based sharding)
- HSDP + AllGather is rejected (would double-shard weights)
regular loading only when the model is outside the DLO+AllGather mmap path;
online quantization with DLO+AllGather is rejected. Use
`--dlo-no-use-allgather` or disable online quantization.
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path because TP-aware
weight-loader callbacks are bypassed. The no-AllGather path retains the
standard TP loader and is experimental.
- HSDP + AllGather is rejected because it would double-shard parameters. HSDP
with no-AllGather is accepted at configuration level, but has limited
end-to-end validation.
- `num_inference_steps=None` is not allowed in DP multi-concurrency mode
**Module Discovery**
@@ -338,7 +349,7 @@ OffloadBackend (base class)
├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
│ (single-GPU, full weights on host)
└── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
(multi-GPU, 1/dp_size sharded weights + AllGather)
(multi-GPU, sharded AllGather or rank-local mode)
```
Factory function `get_offload_backend()` selects the appropriate backend based on
@@ -369,4 +380,10 @@ reasoner/generator components inside the model forward pass.
**Notes:**
- Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
- Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
- Distributed Layerwise Offloading works with any model that supports Layerwise Offloading — no additional model changes required. See [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md) for usage examples.
- Distributed Layerwise Offloading reuses the layerwise block topology, but the
model and parallelism combination still needs validation. AllGather mode
additionally requires a compatible checkpoint/mmap layout; no-AllGather mode
retains the standard loader's rank-local tensors. See the
[DLO design](../../design/feature/distributed_layerwise_offload.md) and
[Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
for usage examples.
+18 -10
View File
@@ -56,10 +56,14 @@ vllm serve /path/to/Cosmos3-Super \
--data-parallel-size 4
```
### Without AllGather (full weights per rank, no sharding)
### Without DLO AllGather (rank-local weights, no DLO sharding)
Use `--dlo-no-use-allgather` when you want each rank to load full weights independently —
no AllGather synchronization, no concurrent request requirement, but N× host memory.
Use `--dlo-no-use-allgather` when you want each rank to stream the standard
loader's rank-local weights independently — no DLO AllGather synchronization,
no concurrent request requirement, but no DLO weight sharding and higher host
memory use. In pure DP, this is equivalent to independent full-weight
layerwise-offload replicas; with TP or HSDP, each rank retains the standard
loader's local tensor layout.
```bash
vllm serve /path/to/Cosmos3-Nano \
@@ -154,9 +158,9 @@ wait
| Flag | Description | Default |
|------|-------------|---------|
| `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` |
| `--data-parallel-size N` | Number of DP ranks (weight sharding + concurrent requests) | `1` |
| `--data-parallel-size N` | Number of DP ranks; DLO weight sharding + concurrent requests when AllGather is enabled | `1` |
| `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` |
| `--dlo-no-use-allgather` | Each rank loads full weights independently (no sharding, no AllGather) | `false` |
| `--dlo-no-use-allgather` | Stream standard-loader rank-local weights independently (no DLO sharding, no DLO AllGather) | `false` |
| `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` |
## Mode comparison
@@ -166,7 +170,7 @@ wait
| DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences |
| DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory |
| DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) |
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Full model | 2 blocks | 0.4× HSDP | No AllGather overhead, high CPU |
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Standard-loader rank-local weights | 2 blocks | 0.4× HSDP | No DLO AllGather overhead, high CPU |
| HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM |
## Memory expectations
@@ -185,7 +189,11 @@ wait
- `--dlo-use-allgather` (default) requires all concurrent requests to have the same
`num_inference_steps` (AllGather is a collective that needs all ranks synchronized)
- Online quantization (FP8) is incompatible with mmap loading — falls back to regular
`load_weights()` automatically
- Tensor Parallel is not supported (DLO uses DP-based sharding, a different dimension)
- HSDP + AllGather is rejected (would double-shard weights)
- Online quantization (FP8) is incompatible with the DLO+AllGather mmap path;
use `--dlo-no-use-allgather` or disable online quantization
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path. TP with
`--dlo-no-use-allgather` retains the standard TP loader and should be treated
as an experimental compatibility path
- HSDP + AllGather is rejected (would double-shard weights). HSDP with
`--dlo-no-use-allgather` is accepted at configuration level, but is not yet a
broadly validated production combination
+41
View File
@@ -2,6 +2,9 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Unit tests for LayerNorm and RMSNorm custom ops in diffusion layers."""
import sys
import types
import pytest
import torch
from pytest_mock import MockerFixture
@@ -299,6 +302,44 @@ def test_rmsnorm_numerical_correctness():
torch.testing.assert_close(out, expected, atol=1e-5, rtol=1e-5)
def test_rmsnorm_dtype_matches_parameter_and_fp32_accumulation_contract():
"""dtype controls gamma storage; reduced-precision inputs still use fp32 accumulation."""
from vllm_omni.diffusion.layers.norm import RMSNorm
hidden_size = 64
eps = 1e-6
norm = RMSNorm(hidden_size, eps=eps, dtype=torch.bfloat16)
x = torch.randn(2, 4, hidden_size, dtype=torch.bfloat16)
assert norm.weight.dtype == torch.bfloat16
x_fp32 = x.float()
expected = (x_fp32 * torch.rsqrt(x_fp32.square().mean(-1, keepdim=True) + eps) * norm.weight.float()).to(x.dtype)
torch.testing.assert_close(norm.forward_native(x), expected, atol=0, rtol=0)
def test_rmsnorm_npu_receives_gamma_with_configured_dtype(monkeypatch: pytest.MonkeyPatch):
"""The NPU fused op receives the bf16 checkpoint gamma used by H3."""
from vllm_omni.diffusion.layers.norm import RMSNorm
captured: dict[str, torch.Tensor] = {}
def npu_rms_norm(x: torch.Tensor, gamma: torch.Tensor, epsilon: float):
captured["gamma"] = gamma
captured["epsilon"] = torch.tensor(epsilon)
return (x,)
monkeypatch.setitem(sys.modules, "torch_npu", types.SimpleNamespace(npu_rms_norm=npu_rms_norm))
norm = RMSNorm(64, eps=1e-5, dtype=torch.bfloat16)
x = torch.randn(2, 4, 64, dtype=torch.bfloat16)
assert norm.forward_npu(x) is x
assert captured["gamma"] is norm.weight
assert captured["gamma"].dtype == torch.bfloat16
assert captured["epsilon"].item() == pytest.approx(1e-5)
# ── RMSNorm compile-path regression tests ──
+34 -1
View File
@@ -10,10 +10,13 @@ forward_hip, and apply_rotary_emb_mindiesd paths.
from __future__ import annotations
import sys
import types
import pytest
import torch
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
from vllm_omni.diffusion.layers.rope import RotaryEmbedding, apply_rotary_emb_mindiesd
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -234,3 +237,33 @@ class TestRotaryEmbeddingNativeShapeRegression:
output = rope.forward_native(x, cos, sin)
assert output.shape == x.shape
assert not torch.isnan(output).any()
def test_mindie_full_dim_rope_keeps_h3_cos_sin_layout(monkeypatch: pytest.MonkeyPatch) -> None:
"""H3 passes complete rotary dims to MindIE instead of the CUDA half-dim layout."""
captured: dict[str, object] = {}
def rotary_position_embedding(x, cos, sin, **kwargs):
captured.update(x=x, cos=cos, sin=sin, kwargs=kwargs)
return x
monkeypatch.setitem(
sys.modules,
"mindiesd",
types.SimpleNamespace(rotary_position_embedding=rotary_position_embedding),
)
x = torch.randn(11, 3, 96, dtype=torch.bfloat16)
cos = torch.randn(11, 96, dtype=torch.bfloat16)
sin = torch.randn(11, 96, dtype=torch.bfloat16)
actual = apply_rotary_emb_mindiesd(x, cos, sin, interleaved=False, half_head_dim=False)
assert actual is x
assert captured["x"] is x
assert captured["cos"] is cos
assert captured["sin"] is sin
assert captured["kwargs"] == {
"rotated_mode": "rotated_half",
"head_first": False,
"fused": True,
}
@@ -61,6 +61,33 @@ def test_packed_attention_is_a_regional_compile_boundary():
assert getattr(MiniMaxH3Attention._run_packed_attention, "_torchdynamo_disable", False)
def test_h3_fused_rope_matches_reference_and_preserves_unrotated_dims():
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
from vllm_omni.diffusion.models.minimax_h3.minimax_h3_transformer import (
MiniMaxH3Attention,
)
attention = object.__new__(MiniMaxH3Attention)
nn.Module.__init__(attention)
attention.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
attention.rope._forward_method = attention.rope.forward_native
x = torch.randn(11, 3, 128, dtype=torch.bfloat16)
freqs_half = torch.randn(11, 48)
freqs = torch.cat((freqs_half, freqs_half), dim=-1)
actual = attention._apply_rope(x, freqs)
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1)
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
x_rot = x[..., :96]
x1, x2 = x_rot.chunk(2, dim=-1)
expected_rot = x_rot * cos + torch.cat((-x2, x1), dim=-1) * sin
expected = torch.cat((expected_rot, x[..., 96:]), dim=-1)
torch.testing.assert_close(actual, expected, atol=0, rtol=0)
torch.testing.assert_close(actual[..., 96:], x[..., 96:], atol=0, rtol=0)
@pytest.mark.parametrize(
("tp_size", "message"),
[
@@ -135,6 +135,7 @@ def _generate_joint_output(omni, config: QualityTestConfig):
num_frames=config.num_frames,
guidance_scale=config.guidance_scale,
sigmas=config.sigmas,
extra_args={"aspect_ratio": "16:9"},
),
)
main_process_peak_mem = torch.accelerator.max_memory_allocated() / (1024**3)
+2 -2
View File
@@ -65,9 +65,9 @@ class LayerNorm(nn.LayerNorm, CustomOp):
class RMSNorm(CustomOp):
def __init__(self, hidden_size: int, eps: float = 1e-6) -> None:
def __init__(self, hidden_size: int, eps: float = 1e-6, dtype: torch.dtype = torch.float32) -> None:
super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size))
self.weight = nn.Parameter(torch.ones(hidden_size, dtype=dtype))
self.variance_epsilon = eps
def _forward_fused(self, x: torch.Tensor) -> torch.Tensor:
+21 -13
View File
@@ -89,10 +89,11 @@ class RotaryEmbedding(CustomOp):
of 1st half and 2nd half (GPT-NeoX style).
"""
def __init__(self, is_neox_style: bool = False) -> None:
def __init__(self, is_neox_style: bool = False, half_head_dim: bool = True) -> None:
super().__init__()
self.is_neox_style = is_neox_style
self.interleaved = not is_neox_style
self.half_head_dim = half_head_dim
self.apply_rotary_emb_flash_attn = None
self.has_mindie = False
# ``find_spec("flash_attn")`` is True as long as *any* package publishes
@@ -110,6 +111,21 @@ class RotaryEmbedding(CustomOp):
if find_spec("mindiesd") is not None:
self.has_mindie = True
def _prepare_half_head_dim_cos_sin(
self,
cos: torch.Tensor,
sin: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Normalize shared cos/sin to the half-dim layout used by CUDA/native kernels."""
if cos.dim() == 3:
# All batch elements share the same rotary position encoding.
cos = cos[0]
sin = sin[0]
if not self.half_head_dim:
cos = cos[..., ::2] if self.interleaved else cos[..., : cos.shape[-1] // 2]
sin = sin[..., ::2] if self.interleaved else sin[..., : sin.shape[-1] // 2]
return cos, sin
def forward_cuda(
self,
x: torch.Tensor,
@@ -118,10 +134,7 @@ class RotaryEmbedding(CustomOp):
) -> torch.Tensor:
from vllm.vllm_flash_attn.layers.rotary import apply_rotary_emb
if cos.dim() == 3:
# (B, S, D/2) -> (S, D/2)
cos = cos[0]
sin = sin[0]
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
x, squeezed = _ensure_batch_dim(x)
output = apply_rotary_emb(
@@ -141,10 +154,7 @@ class RotaryEmbedding(CustomOp):
if self.apply_rotary_emb_flash_attn is None:
return self.forward_cuda(x, cos, sin)
if cos.dim() == 3:
# (B, S, D/2) -> (S, D/2)
cos = cos[0]
sin = sin[0]
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
x, squeezed = _ensure_batch_dim(x)
output = self.apply_rotary_emb_flash_attn(
@@ -162,7 +172,7 @@ class RotaryEmbedding(CustomOp):
sin: torch.Tensor,
) -> torch.Tensor:
if self.has_mindie:
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved)
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved, self.half_head_dim)
else:
return self.forward_native(x, cos, sin)
@@ -191,9 +201,7 @@ class RotaryEmbedding(CustomOp):
# All batch elements share the same rotary position encoding.
# Strip the batch dim so the underlying op broadcasts over the batch,
# consistent with forward_cuda / forward_hip / apply_rotary_emb_mindiesd.
if cos.dim() == 3:
cos = cos[0]
sin = sin[0]
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
return apply_rotary_emb_torch(
x,
cos,
@@ -33,6 +33,8 @@ from vllm_omni.diffusion.distributed.sp_plan import (
SequenceParallelInput,
SequenceParallelOutput,
)
from vllm_omni.diffusion.layers.norm import RMSNorm
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
if TYPE_CHECKING:
from vllm.model_executor.layers.quantization.base_config import (
@@ -168,16 +170,11 @@ def _reorder_grouped_qkv_to_qkv(
)
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> nn.RMSNorm:
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> RMSNorm:
# RMSNorm uses fp32 accumulation with bf16 inputs and outputs.
# torch.nn.RMSNorm upcasts reduced-precision inputs for the variance
# reduction, matching that accumulation semantic.
return nn.RMSNorm(size, eps=eps, dtype=dtype)
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
x1, x2 = torch.chunk(x, 2, dim=-1)
return torch.cat((-x2, x1), dim=-1)
return RMSNorm(size, eps=eps, dtype=dtype)
def _modulate_scale_shift(
@@ -230,20 +227,6 @@ class MiniMaxH3Rope(nn.Module):
return torch.cat((half, half), dim=-1) # [S, 96]
def _apply_rope(x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
"""Rotate the first rot_dim head dims; pass the rest through.
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
are cast to the activation dtype before the elementwise math.
"""
rot_dim = freqs.shape[-1]
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1) # [T, 1, rot_dim]
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
x_rot = (x_rot * cos) + (_rotate_half(x_rot) * sin)
return torch.cat((x_rot, x_pass), dim=-1)
class MiniMaxH3TimeEmbedder(nn.Module):
def __init__(
self,
@@ -349,6 +332,7 @@ class MiniMaxH3Attention(nn.Module):
self.num_kv_heads = self.qkv_proj.num_kv_heads
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
self.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
self.out_proj = RowParallelLinear(
inner_dim,
arch.hidden_size,
@@ -367,6 +351,19 @@ class MiniMaxH3Attention(nn.Module):
skip_sequence_parallel=skip_sequence_parallel,
)
def _apply_rope(self, x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
"""Rotate the first rot_dim head dims; pass the rest through.
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
are cast to the activation dtype before the elementwise math.
"""
rot_dim = freqs.shape[-1]
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
cos = torch.cos(freqs).to(x.dtype) # [T, rot_dim]
sin = torch.sin(freqs).to(x.dtype)
x_rot = self.rope(x_rot, cos, sin)
return torch.cat((x_rot, x_pass), dim=-1)
@torch.compiler.disable
def _run_packed_attention(
self,
@@ -450,8 +447,8 @@ class MiniMaxH3Attention(nn.Module):
q = self.q_norm(q)
k = self.k_norm(k)
if rope_freqs is not None:
q = _apply_rope(q, rope_freqs)
k = _apply_rope(k, rope_freqs)
q = self._apply_rope(q, rope_freqs)
k = self._apply_rope(k, rope_freqs)
# The packed layout uses a second document for alignment padding.
# Local/Ulysses backends unpad it, while Ring keeps aligned rows for
+2 -2
View File
@@ -193,7 +193,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
self.to(torch.float32)
devices = [parameter.device] if parameter.device.type != "cpu" else []
try:
with torch.random.fork_rng(devices=devices):
with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
for device in devices:
with self.device_module.device(device):
@@ -237,7 +237,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
self.to(torch.float32)
devices = [parameter.device] if parameter.device.type != "cpu" else []
try:
with torch.random.fork_rng(devices=devices):
with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
for device in devices:
with self.device_module.device(device):