Compare commits

..
14 changed files with 377 additions and 154 deletions

No files matched your search

+23 -41
View File
@@ -101,49 +101,31 @@ nav:
- CI: contributing/ci - CI: contributing/ci
- Design Documents: - Design Documents:
- design/index.md - design/index.md
- Architecture Documents: - design/architecture_overview.md
- Architecture Overview: design/architecture_overview.md
- Feature Design: - Feature Design:
- Runtime and Stage Execution: - design/feature/disaggregated_inference.md
- Disaggregated Inference: design/feature/disaggregated_inference.md - design/feature/ray_based_execution.md
- Ray-based Execution: design/feature/ray_based_execution.md - design/feature/omni_connectors/
- Async Chunk: design/feature/async_chunk.md - design/feature/prefix_caching.md
- Async Diffusion Output: design/feature/async_diffusion_output.md - design/feature/cfg_parallel.md
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md - design/feature/expert_parallel.md
- Automatic Prefix Caching: design/feature/prefix_caching.md - design/feature/sequence_parallel.md
- Communication: - design/feature/tensor_parallel.md
- OmniConnector Implementations: - design/feature/vae_parallel.md
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md - design/feature/hsdp.md
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md - design/feature/cache_dit.md
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md - design/feature/teacache.md
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md - design/feature/skip_softmax.md
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md - design/feature/async_chunk.md
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md - design/feature/omni_async_output_materialization.md
- Diffusion Acceleration: - design/feature/vae_parallel.md
- Parallelism: - design/feature/diffusion_continuous_batching.md
- CFG-Parallel: design/feature/cfg_parallel.md - design/feature/distributed_layerwise_offload.md
- Expert Parallel: design/feature/expert_parallel.md
- Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md
- Pipeline Parallel: design/feature/pipeline_parallel.md
- Sequence Parallel: design/feature/sequence_parallel.md
- Tensor Parallel: design/feature/tensor_parallel.md
- VAE Patch Parallelism: design/feature/vae_parallel.md
- Attention Backends:
- Skip-Softmax: design/feature/skip_softmax.md
- Quantization: user_guide/quantization/overview.md
- Cache-DiT: design/feature/cache_dit.md
- TeaCache: design/feature/teacache.md
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
- Infrastructure and Performance:
- Prometheus Metrics: design/metrics.md
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
- Module Design: - Module Design:
- Runtime Modules: - design/module/ar_module.md
- AR Module: design/module/ar_module.md - design/module/dit_module.md
- DIT Module: design/module/dit_module.md - design/module/entrypoint_module.md
- Orchestration: - design/module/async_omni_architecture.md
- Entrypoint Module: design/module/entrypoint_module.md
- AsyncOmni Architecture: design/module/async_omni_architecture.md
- Docs Guide: contributing/DOCS_GUIDE.md - Docs Guide: contributing/DOCS_GUIDE.md
- API Reference: - API Reference:
- api/README.md - api/README.md
@@ -0,0 +1,156 @@
# Distributed Layerwise Offload
This document describes distributed layerwise offload (DLO) for diffusion
models. DLO keeps only a small number of DiT blocks on the accelerator and
streams the remaining blocks from host memory. The distributed backend can
either shard those host-side weights across an existing parallel group or keep
the standard loader's rank-local weights and avoid an additional collective.
For user-facing commands, see the
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload_diffusion.md)
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
## Status
DLO is implemented for multi-device diffusion execution. The default
AllGather path is the primary path for DP and SP deployments. The
`--dlo-no-use-allgather` path is a rank-local compatibility mode: it is useful
for standard-loader sharding, workstation bring-up, and systems where an
additional DLO collective is undesirable, but it does not reduce host weight
storage across ranks.
The compatibility matrix below describes the current implementation. The
unit-level guards are covered, but not every parallelism combination has a
full model-and-hardware end-to-end test.
## Design
### DLO consumes the existing parallel topology
DLO does not create a new DP, TP, or SP topology. It reads the configured
`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
standard distributed groups have been initialized.
The DLO weight-sharding group is selected as follows:
1. Use the existing DP group when `data_parallel_size > 1`.
2. When DP is one and SP is greater than one, use the SP group.
3. Otherwise, run rank-locally without a DLO process group.
TP is deliberately not used as DLO's AllGather group. HSDP has its own
parameter-sharding lifecycle and is not allowed to be sharded a second time by
DLO's AllGather path.
### AllGather path
With the default `dlo_use_allgather=True`, each rank stores approximately
`1 / group_size` of each streamable block in pinned host memory. The next
block's shard is copied to a device buffer and reconstructed with
`all_gather_into_tensor` on a communication stream while the current block is
executing.
```text
Compute: [Block N] [Block N+1] [Block N+2]
H2D: [shard N+1] [shard N+2]
AllGather: [full N+1] [full N+2]
Buffers: [current slot] [prefetch slot] [current slot]
```
![DLO double-buffer prefetch pipeline](../figures/dlo/dlo_pipeline.gif)
The backend uses two shared device buffers, so accelerator weight residency is
bounded by the largest streamed blocks rather than the complete model.
When DP is greater than one, the engine can process one request per DP rank in
the same denoising wave. Because AllGather is a collective, all participating
requests must take the same execution path at every denoising step.
### Rank-local path without DLO AllGather
With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
one. The regular model loader remains responsible for preparing each rank's
weights, including TP-local tensors or HSDP-managed parameters. DLO then
streams those rank-local tensors block by block using H2D copies only.
This mode means:
- DP still provides independent replicas, but DLO does not shard weights
across DP ranks.
- SP still performs its normal activation/attention collectives, but DLO does
not shard weights across SP ranks.
- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
DLO's additional weight AllGather is disabled.
- Pure DP deployments keep a full host-side model copy per rank, subject to
shared OS page-cache behavior.
- The scheduler does not require a synchronized DP request wave for DLO.
This path is intentionally not implemented by reusing the AllGather mmap
loader. It relies on the standard loader so model-specific TP/HSDP transforms
remain intact.
## Parallelism compatibility
| Parallelism | DLO + AllGather | DLO without AllGather |
|---|---|---|
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. DP replicas remain independent; no DLO weight sharding or cross-DP DLO collective. |
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
| **TP > 1** | Unsupported in the DLO mmap path. TP-aware loader callbacks are bypassed, so the backend rejects this configuration when it enters that path. | Standard loading is retained and TP-local tensors can be streamed. This is the intended compatibility path, but it still needs broader model and hardware validation. |
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
### Combined dimensions
- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
than one; SP continues to use its own sequence-parallel group. If DP is one,
the SP group becomes DLO's sharding group in AllGather mode.
- **DP + TP/SP without AllGather:** standard model loading defines the
rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
collective.
- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
materialization and synchronization.
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
configuration.
## Request and loading constraints
AllGather DP multi-concurrency requires:
- explicit `num_inference_steps`;
- the same `num_inference_steps` for all requests in a wave; and
- identical request arguments that affect the collective execution path.
The no-AllGather path does not impose these DLO-specific synchronized-wave
requirements.
The mmap loader is used only by the supported DLO+AllGather path when the
model has a compatible checkpoint layout. Online quantization is incompatible
with that sharded mmap path; use `--dlo-no-use-allgather` or disable online
quantization. Models that do not meet the mmap requirements use the regular
loader path.
## Validation coverage
Current source-level validation includes:
- HSDP + DLO + AllGather rejection;
- HSDP + DLO without AllGather acceptance at configuration level;
- TP rejection in the DLO+AllGather mmap path;
- resident-layer requests requiring no-AllGather;
- DP request-wave validation for denoising-step compatibility;
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
tests.
The highest-value missing coverage is end-to-end numerical comparison against
ordinary layerwise offload for DP+SP, TP+no-AllGather, and HSDP+SP+no-AllGather
on the target CUDA/NCCL or CANN/HCCL hardware.
## Recommendations
- Use **DP + DLO AllGather** for the supported throughput and host-memory
scaling path.
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
not the goal.
- Use **no-AllGather** to bring up TP or workstation PCIe configurations, with
the expectation of higher host-memory use and lower validation confidence.
- Prefer **HSDP alone** for production HSDP deployments until the combined
HSDP + DLO no-AllGather path has broader end-to-end coverage.
Binary file not shown.

After

Width:  |  Height:  |  Size: 195 KiB

+3 -50
View File
@@ -1,8 +1,6 @@
# Design Documents # Design Documents
This section contains design documents and architecture specifications for This section contains design documents and architecture specifications for vLLM-Omni.
vLLM-Omni. The sidebar groups documents by the system concern they describe,
while the source paths remain stable so existing links continue to work.
## Architecture Documents ## Architecture Documents
@@ -10,63 +8,18 @@ while the source paths remain stable so existing links continue to work.
## Feature Design Documents ## Feature Design Documents
### Runtime and stage execution
- [Disaggregated Inference](feature/disaggregated_inference.md) - [Disaggregated Inference](feature/disaggregated_inference.md)
- [Ray-based Execution](feature/ray_based_execution.md) - [Ray-based Execution](feature/ray_based_execution.md)
- [Async Chunk](feature/async_chunk.md)
- [Async Diffusion Output](feature/async_diffusion_output.md)
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md) - [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
### Communication
#### OmniConnector implementations
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
### Diffusion acceleration
#### Parallelism
- [CFG-Parallel](feature/cfg_parallel.md)
- [Expert Parallel](feature/expert_parallel.md)
- [Hybrid Sharded Data Parallel (HSDP)](feature/hsdp.md)
- [Pipeline Parallel](feature/pipeline_parallel.md)
- [Sequence Parallel](feature/sequence_parallel.md)
- [Tensor Parallel](feature/tensor_parallel.md)
- [VAE Patch Parallelism](feature/vae_parallel.md)
#### Attention Backends
- [Skip-Softmax](feature/skip_softmax.md)
#### Quantization
- [Quantization Overview](../user_guide/quantization/overview.md)
- [Cache-DiT](feature/cache_dit.md)
- [TeaCache](feature/teacache.md)
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md) - [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
## Infrastructure and Performance ## Infrastructure Design Documents
- [Prometheus Metrics](metrics.md) - [Prometheus Metrics](metrics.md)
- [Speech Generation Performance Optimizations](qwen3_omni_tts_performance_optimization.md)
## Module Design Documents ## Module Design Documents
### Runtime Modules
- [AR Module](module/ar_module.md) - [AR Module](module/ar_module.md)
- [DIT Module](module/dit_module.md) - [DIT Module](module/dit_module.md)
### Orchestration
- [Entrypoint Module](module/entrypoint_module.md) - [Entrypoint Module](module/entrypoint_module.md)
- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md)
@@ -184,10 +184,15 @@ class Flux2Transformer2DModel(nn.Module):
### How It Works ### How It Works
Distributed layerwise offloading extends single-GPU layerwise offloading to Distributed layerwise offloading extends single-GPU layerwise offloading to
multi-device deployments. Each DP rank stores only **1/dp_size** of the model multi-device deployments. In the default mode, each rank stores only
weights in host memory; full layer weights are reconstructed at runtime via **1/dp_size** of the model weights in host memory; full layer weights are
**AllGather** on a dedicated communication stream, overlapped with computation reconstructed at runtime via **AllGather** on a dedicated communication stream,
via a fixed double-buffer scheme. overlapped with computation via a fixed double-buffer scheme. The
`--dlo-no-use-allgather` mode instead streams standard-loader rank-local
weights independently and does not shard weights across ranks.
For the implementation design and compatibility matrix, see
[Distributed Layerwise Offload](../../design/feature/distributed_layerwise_offload.md).
**Key features:** **Key features:**
- **Weight sharding + AllGather**: each rank stores 1/dp_size of weights, - **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
@@ -219,7 +224,7 @@ vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \ --enable-distributed-layerwise-offload \
--data-parallel-size 4 --data-parallel-size 4
# Without AllGather (each rank loads full weights, no sharding) # Without DLO AllGather (each rank streams standard-loader rank-local weights)
vllm serve /path/to/model --omni \ vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \ --enable-distributed-layerwise-offload \
--data-parallel-size 4 \ --data-parallel-size 4 \
@@ -247,9 +252,9 @@ m = Omni(
| Flag | Description | Default | | Flag | Description | Default |
|------|-------------|---------| |------|-------------|---------|
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` | | `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
| `--data-parallel-size N` | Number of DP ranks for weight sharding | `1` | | `--data-parallel-size N` | Number of DP ranks; DLO weight-sharding group when AllGather is enabled | `1` |
| `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` | | `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
| `--dlo-no-use-allgather` | Full weights per rank (no sharding, no AllGather) | `false` | | `--dlo-no-use-allgather` | Use standard-loader rank-local weights; no DLO sharding or AllGather | `false` |
### How model weights are loaded (mmap path) ### How model weights are loaded (mmap path)
@@ -303,9 +308,15 @@ request while AllGather synchronizes only weight shards (request-independent).
### Limitations ### Limitations
- Online quantization (FP8) is incompatible with mmap loading — falls back to - Online quantization (FP8) is incompatible with mmap loading — falls back to
regular `load_weights()` automatically regular loading only when the model is outside the DLO+AllGather mmap path;
- Tensor Parallel is not supported (DLO uses DP-based sharding) online quantization with DLO+AllGather is rejected. Use
- HSDP + AllGather is rejected (would double-shard weights) `--dlo-no-use-allgather` or disable online quantization.
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path because TP-aware
weight-loader callbacks are bypassed. The no-AllGather path retains the
standard TP loader and is experimental.
- HSDP + AllGather is rejected because it would double-shard parameters. HSDP
with no-AllGather is accepted at configuration level, but has limited
end-to-end validation.
- `num_inference_steps=None` is not allowed in DP multi-concurrency mode - `num_inference_steps=None` is not allowed in DP multi-concurrency mode
**Module Discovery** **Module Discovery**
@@ -338,7 +349,7 @@ OffloadBackend (base class)
├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook ├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
│ (single-GPU, full weights on host) │ (single-GPU, full weights on host)
└── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook └── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
(multi-GPU, 1/dp_size sharded weights + AllGather) (multi-GPU, sharded AllGather or rank-local mode)
``` ```
Factory function `get_offload_backend()` selects the appropriate backend based on Factory function `get_offload_backend()` selects the appropriate backend based on
@@ -369,4 +380,10 @@ reasoner/generator components inside the model forward pass.
**Notes:** **Notes:**
- Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally - Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
- Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks - Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
- Distributed Layerwise Offloading works with any model that supports Layerwise Offloading — no additional model changes required. See [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md) for usage examples. - Distributed Layerwise Offloading reuses the layerwise block topology, but the
model and parallelism combination still needs validation. AllGather mode
additionally requires a compatible checkpoint/mmap layout; no-AllGather mode
retains the standard loader's rank-local tensors. See the
[DLO design](../../design/feature/distributed_layerwise_offload.md) and
[Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
for usage examples.
+18 -10
View File
@@ -56,10 +56,14 @@ vllm serve /path/to/Cosmos3-Super \
--data-parallel-size 4 --data-parallel-size 4
``` ```
### Without AllGather (full weights per rank, no sharding) ### Without DLO AllGather (rank-local weights, no DLO sharding)
Use `--dlo-no-use-allgather` when you want each rank to load full weights independently — Use `--dlo-no-use-allgather` when you want each rank to stream the standard
no AllGather synchronization, no concurrent request requirement, but N× host memory. loader's rank-local weights independently — no DLO AllGather synchronization,
no concurrent request requirement, but no DLO weight sharding and higher host
memory use. In pure DP, this is equivalent to independent full-weight
layerwise-offload replicas; with TP or HSDP, each rank retains the standard
loader's local tensor layout.
```bash ```bash
vllm serve /path/to/Cosmos3-Nano \ vllm serve /path/to/Cosmos3-Nano \
@@ -154,9 +158,9 @@ wait
| Flag | Description | Default | | Flag | Description | Default |
|------|-------------|---------| |------|-------------|---------|
| `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` | | `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` |
| `--data-parallel-size N` | Number of DP ranks (weight sharding + concurrent requests) | `1` | | `--data-parallel-size N` | Number of DP ranks; DLO weight sharding + concurrent requests when AllGather is enabled | `1` |
| `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` | | `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` |
| `--dlo-no-use-allgather` | Each rank loads full weights independently (no sharding, no AllGather) | `false` | | `--dlo-no-use-allgather` | Stream standard-loader rank-local weights independently (no DLO sharding, no DLO AllGather) | `false` |
| `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` | | `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` |
## Mode comparison ## Mode comparison
@@ -166,7 +170,7 @@ wait
| DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences | | DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences |
| DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory | | DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory |
| DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) | | DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) |
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Full model | 2 blocks | 0.4× HSDP | No AllGather overhead, high CPU | | DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Standard-loader rank-local weights | 2 blocks | 0.4× HSDP | No DLO AllGather overhead, high CPU |
| HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM | | HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM |
## Memory expectations ## Memory expectations
@@ -185,7 +189,11 @@ wait
- `--dlo-use-allgather` (default) requires all concurrent requests to have the same - `--dlo-use-allgather` (default) requires all concurrent requests to have the same
`num_inference_steps` (AllGather is a collective that needs all ranks synchronized) `num_inference_steps` (AllGather is a collective that needs all ranks synchronized)
- Online quantization (FP8) is incompatible with mmap loading — falls back to regular - Online quantization (FP8) is incompatible with the DLO+AllGather mmap path;
`load_weights()` automatically use `--dlo-no-use-allgather` or disable online quantization
- Tensor Parallel is not supported (DLO uses DP-based sharding, a different dimension) - Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path. TP with
- HSDP + AllGather is rejected (would double-shard weights) `--dlo-no-use-allgather` retains the standard TP loader and should be treated
as an experimental compatibility path
- HSDP + AllGather is rejected (would double-shard weights). HSDP with
`--dlo-no-use-allgather` is accepted at configuration level, but is not yet a
broadly validated production combination
+41
View File
@@ -2,6 +2,9 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Unit tests for LayerNorm and RMSNorm custom ops in diffusion layers.""" """Unit tests for LayerNorm and RMSNorm custom ops in diffusion layers."""
import sys
import types
import pytest import pytest
import torch import torch
from pytest_mock import MockerFixture from pytest_mock import MockerFixture
@@ -299,6 +302,44 @@ def test_rmsnorm_numerical_correctness():
torch.testing.assert_close(out, expected, atol=1e-5, rtol=1e-5) torch.testing.assert_close(out, expected, atol=1e-5, rtol=1e-5)
def test_rmsnorm_dtype_matches_parameter_and_fp32_accumulation_contract():
"""dtype controls gamma storage; reduced-precision inputs still use fp32 accumulation."""
from vllm_omni.diffusion.layers.norm import RMSNorm
hidden_size = 64
eps = 1e-6
norm = RMSNorm(hidden_size, eps=eps, dtype=torch.bfloat16)
x = torch.randn(2, 4, hidden_size, dtype=torch.bfloat16)
assert norm.weight.dtype == torch.bfloat16
x_fp32 = x.float()
expected = (x_fp32 * torch.rsqrt(x_fp32.square().mean(-1, keepdim=True) + eps) * norm.weight.float()).to(x.dtype)
torch.testing.assert_close(norm.forward_native(x), expected, atol=0, rtol=0)
def test_rmsnorm_npu_receives_gamma_with_configured_dtype(monkeypatch: pytest.MonkeyPatch):
"""The NPU fused op receives the bf16 checkpoint gamma used by H3."""
from vllm_omni.diffusion.layers.norm import RMSNorm
captured: dict[str, torch.Tensor] = {}
def npu_rms_norm(x: torch.Tensor, gamma: torch.Tensor, epsilon: float):
captured["gamma"] = gamma
captured["epsilon"] = torch.tensor(epsilon)
return (x,)
monkeypatch.setitem(sys.modules, "torch_npu", types.SimpleNamespace(npu_rms_norm=npu_rms_norm))
norm = RMSNorm(64, eps=1e-5, dtype=torch.bfloat16)
x = torch.randn(2, 4, 64, dtype=torch.bfloat16)
assert norm.forward_npu(x) is x
assert captured["gamma"] is norm.weight
assert captured["gamma"].dtype == torch.bfloat16
assert captured["epsilon"].item() == pytest.approx(1e-5)
# ── RMSNorm compile-path regression tests ── # ── RMSNorm compile-path regression tests ──
+34 -1
View File
@@ -10,10 +10,13 @@ forward_hip, and apply_rotary_emb_mindiesd paths.
from __future__ import annotations from __future__ import annotations
import sys
import types
import pytest import pytest
import torch import torch
from vllm_omni.diffusion.layers.rope import RotaryEmbedding from vllm_omni.diffusion.layers.rope import RotaryEmbedding, apply_rotary_emb_mindiesd
pytestmark = [pytest.mark.core_model, pytest.mark.cpu] pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -234,3 +237,33 @@ class TestRotaryEmbeddingNativeShapeRegression:
output = rope.forward_native(x, cos, sin) output = rope.forward_native(x, cos, sin)
assert output.shape == x.shape assert output.shape == x.shape
assert not torch.isnan(output).any() assert not torch.isnan(output).any()
def test_mindie_full_dim_rope_keeps_h3_cos_sin_layout(monkeypatch: pytest.MonkeyPatch) -> None:
"""H3 passes complete rotary dims to MindIE instead of the CUDA half-dim layout."""
captured: dict[str, object] = {}
def rotary_position_embedding(x, cos, sin, **kwargs):
captured.update(x=x, cos=cos, sin=sin, kwargs=kwargs)
return x
monkeypatch.setitem(
sys.modules,
"mindiesd",
types.SimpleNamespace(rotary_position_embedding=rotary_position_embedding),
)
x = torch.randn(11, 3, 96, dtype=torch.bfloat16)
cos = torch.randn(11, 96, dtype=torch.bfloat16)
sin = torch.randn(11, 96, dtype=torch.bfloat16)
actual = apply_rotary_emb_mindiesd(x, cos, sin, interleaved=False, half_head_dim=False)
assert actual is x
assert captured["x"] is x
assert captured["cos"] is cos
assert captured["sin"] is sin
assert captured["kwargs"] == {
"rotated_mode": "rotated_half",
"head_first": False,
"fused": True,
}
@@ -61,6 +61,33 @@ def test_packed_attention_is_a_regional_compile_boundary():
assert getattr(MiniMaxH3Attention._run_packed_attention, "_torchdynamo_disable", False) assert getattr(MiniMaxH3Attention._run_packed_attention, "_torchdynamo_disable", False)
def test_h3_fused_rope_matches_reference_and_preserves_unrotated_dims():
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
from vllm_omni.diffusion.models.minimax_h3.minimax_h3_transformer import (
MiniMaxH3Attention,
)
attention = object.__new__(MiniMaxH3Attention)
nn.Module.__init__(attention)
attention.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
attention.rope._forward_method = attention.rope.forward_native
x = torch.randn(11, 3, 128, dtype=torch.bfloat16)
freqs_half = torch.randn(11, 48)
freqs = torch.cat((freqs_half, freqs_half), dim=-1)
actual = attention._apply_rope(x, freqs)
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1)
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
x_rot = x[..., :96]
x1, x2 = x_rot.chunk(2, dim=-1)
expected_rot = x_rot * cos + torch.cat((-x2, x1), dim=-1) * sin
expected = torch.cat((expected_rot, x[..., 96:]), dim=-1)
torch.testing.assert_close(actual, expected, atol=0, rtol=0)
torch.testing.assert_close(actual[..., 96:], x[..., 96:], atol=0, rtol=0)
@pytest.mark.parametrize( @pytest.mark.parametrize(
("tp_size", "message"), ("tp_size", "message"),
[ [
@@ -135,6 +135,7 @@ def _generate_joint_output(omni, config: QualityTestConfig):
num_frames=config.num_frames, num_frames=config.num_frames,
guidance_scale=config.guidance_scale, guidance_scale=config.guidance_scale,
sigmas=config.sigmas, sigmas=config.sigmas,
extra_args={"aspect_ratio": "16:9"},
), ),
) )
main_process_peak_mem = torch.accelerator.max_memory_allocated() / (1024**3) main_process_peak_mem = torch.accelerator.max_memory_allocated() / (1024**3)
+2 -2
View File
@@ -65,9 +65,9 @@ class LayerNorm(nn.LayerNorm, CustomOp):
class RMSNorm(CustomOp): class RMSNorm(CustomOp):
def __init__(self, hidden_size: int, eps: float = 1e-6) -> None: def __init__(self, hidden_size: int, eps: float = 1e-6, dtype: torch.dtype = torch.float32) -> None:
super().__init__() super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size)) self.weight = nn.Parameter(torch.ones(hidden_size, dtype=dtype))
self.variance_epsilon = eps self.variance_epsilon = eps
def _forward_fused(self, x: torch.Tensor) -> torch.Tensor: def _forward_fused(self, x: torch.Tensor) -> torch.Tensor:
+21 -13
View File
@@ -89,10 +89,11 @@ class RotaryEmbedding(CustomOp):
of 1st half and 2nd half (GPT-NeoX style). of 1st half and 2nd half (GPT-NeoX style).
""" """
def __init__(self, is_neox_style: bool = False) -> None: def __init__(self, is_neox_style: bool = False, half_head_dim: bool = True) -> None:
super().__init__() super().__init__()
self.is_neox_style = is_neox_style self.is_neox_style = is_neox_style
self.interleaved = not is_neox_style self.interleaved = not is_neox_style
self.half_head_dim = half_head_dim
self.apply_rotary_emb_flash_attn = None self.apply_rotary_emb_flash_attn = None
self.has_mindie = False self.has_mindie = False
# ``find_spec("flash_attn")`` is True as long as *any* package publishes # ``find_spec("flash_attn")`` is True as long as *any* package publishes
@@ -110,6 +111,21 @@ class RotaryEmbedding(CustomOp):
if find_spec("mindiesd") is not None: if find_spec("mindiesd") is not None:
self.has_mindie = True self.has_mindie = True
def _prepare_half_head_dim_cos_sin(
self,
cos: torch.Tensor,
sin: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Normalize shared cos/sin to the half-dim layout used by CUDA/native kernels."""
if cos.dim() == 3:
# All batch elements share the same rotary position encoding.
cos = cos[0]
sin = sin[0]
if not self.half_head_dim:
cos = cos[..., ::2] if self.interleaved else cos[..., : cos.shape[-1] // 2]
sin = sin[..., ::2] if self.interleaved else sin[..., : sin.shape[-1] // 2]
return cos, sin
def forward_cuda( def forward_cuda(
self, self,
x: torch.Tensor, x: torch.Tensor,
@@ -118,10 +134,7 @@ class RotaryEmbedding(CustomOp):
) -> torch.Tensor: ) -> torch.Tensor:
from vllm.vllm_flash_attn.layers.rotary import apply_rotary_emb from vllm.vllm_flash_attn.layers.rotary import apply_rotary_emb
if cos.dim() == 3: cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
# (B, S, D/2) -> (S, D/2)
cos = cos[0]
sin = sin[0]
x, squeezed = _ensure_batch_dim(x) x, squeezed = _ensure_batch_dim(x)
output = apply_rotary_emb( output = apply_rotary_emb(
@@ -141,10 +154,7 @@ class RotaryEmbedding(CustomOp):
if self.apply_rotary_emb_flash_attn is None: if self.apply_rotary_emb_flash_attn is None:
return self.forward_cuda(x, cos, sin) return self.forward_cuda(x, cos, sin)
if cos.dim() == 3: cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
# (B, S, D/2) -> (S, D/2)
cos = cos[0]
sin = sin[0]
x, squeezed = _ensure_batch_dim(x) x, squeezed = _ensure_batch_dim(x)
output = self.apply_rotary_emb_flash_attn( output = self.apply_rotary_emb_flash_attn(
@@ -162,7 +172,7 @@ class RotaryEmbedding(CustomOp):
sin: torch.Tensor, sin: torch.Tensor,
) -> torch.Tensor: ) -> torch.Tensor:
if self.has_mindie: if self.has_mindie:
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved) return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved, self.half_head_dim)
else: else:
return self.forward_native(x, cos, sin) return self.forward_native(x, cos, sin)
@@ -191,9 +201,7 @@ class RotaryEmbedding(CustomOp):
# All batch elements share the same rotary position encoding. # All batch elements share the same rotary position encoding.
# Strip the batch dim so the underlying op broadcasts over the batch, # Strip the batch dim so the underlying op broadcasts over the batch,
# consistent with forward_cuda / forward_hip / apply_rotary_emb_mindiesd. # consistent with forward_cuda / forward_hip / apply_rotary_emb_mindiesd.
if cos.dim() == 3: cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
cos = cos[0]
sin = sin[0]
return apply_rotary_emb_torch( return apply_rotary_emb_torch(
x, x,
cos, cos,
@@ -33,6 +33,8 @@ from vllm_omni.diffusion.distributed.sp_plan import (
SequenceParallelInput, SequenceParallelInput,
SequenceParallelOutput, SequenceParallelOutput,
) )
from vllm_omni.diffusion.layers.norm import RMSNorm
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
if TYPE_CHECKING: if TYPE_CHECKING:
from vllm.model_executor.layers.quantization.base_config import ( from vllm.model_executor.layers.quantization.base_config import (
@@ -168,16 +170,11 @@ def _reorder_grouped_qkv_to_qkv(
) )
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> nn.RMSNorm: def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> RMSNorm:
# RMSNorm uses fp32 accumulation with bf16 inputs and outputs. # RMSNorm uses fp32 accumulation with bf16 inputs and outputs.
# torch.nn.RMSNorm upcasts reduced-precision inputs for the variance # torch.nn.RMSNorm upcasts reduced-precision inputs for the variance
# reduction, matching that accumulation semantic. # reduction, matching that accumulation semantic.
return nn.RMSNorm(size, eps=eps, dtype=dtype) return RMSNorm(size, eps=eps, dtype=dtype)
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
x1, x2 = torch.chunk(x, 2, dim=-1)
return torch.cat((-x2, x1), dim=-1)
def _modulate_scale_shift( def _modulate_scale_shift(
@@ -230,20 +227,6 @@ class MiniMaxH3Rope(nn.Module):
return torch.cat((half, half), dim=-1) # [S, 96] return torch.cat((half, half), dim=-1) # [S, 96]
def _apply_rope(x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
"""Rotate the first rot_dim head dims; pass the rest through.
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
are cast to the activation dtype before the elementwise math.
"""
rot_dim = freqs.shape[-1]
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1) # [T, 1, rot_dim]
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
x_rot = (x_rot * cos) + (_rotate_half(x_rot) * sin)
return torch.cat((x_rot, x_pass), dim=-1)
class MiniMaxH3TimeEmbedder(nn.Module): class MiniMaxH3TimeEmbedder(nn.Module):
def __init__( def __init__(
self, self,
@@ -349,6 +332,7 @@ class MiniMaxH3Attention(nn.Module):
self.num_kv_heads = self.qkv_proj.num_kv_heads self.num_kv_heads = self.qkv_proj.num_kv_heads
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps) self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps) self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
self.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
self.out_proj = RowParallelLinear( self.out_proj = RowParallelLinear(
inner_dim, inner_dim,
arch.hidden_size, arch.hidden_size,
@@ -367,6 +351,19 @@ class MiniMaxH3Attention(nn.Module):
skip_sequence_parallel=skip_sequence_parallel, skip_sequence_parallel=skip_sequence_parallel,
) )
def _apply_rope(self, x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
"""Rotate the first rot_dim head dims; pass the rest through.
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
are cast to the activation dtype before the elementwise math.
"""
rot_dim = freqs.shape[-1]
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
cos = torch.cos(freqs).to(x.dtype) # [T, rot_dim]
sin = torch.sin(freqs).to(x.dtype)
x_rot = self.rope(x_rot, cos, sin)
return torch.cat((x_rot, x_pass), dim=-1)
@torch.compiler.disable @torch.compiler.disable
def _run_packed_attention( def _run_packed_attention(
self, self,
@@ -450,8 +447,8 @@ class MiniMaxH3Attention(nn.Module):
q = self.q_norm(q) q = self.q_norm(q)
k = self.k_norm(k) k = self.k_norm(k)
if rope_freqs is not None: if rope_freqs is not None:
q = _apply_rope(q, rope_freqs) q = self._apply_rope(q, rope_freqs)
k = _apply_rope(k, rope_freqs) k = self._apply_rope(k, rope_freqs)
# The packed layout uses a second document for alignment padding. # The packed layout uses a second document for alignment padding.
# Local/Ulysses backends unpad it, while Ring keeps aligned rows for # Local/Ulysses backends unpad it, while Ring keeps aligned rows for
+2 -2
View File
@@ -193,7 +193,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
self.to(torch.float32) self.to(torch.float32)
devices = [parameter.device] if parameter.device.type != "cpu" else [] devices = [parameter.device] if parameter.device.type != "cpu" else []
try: try:
with torch.random.fork_rng(devices=devices): with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED) torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
for device in devices: for device in devices:
with self.device_module.device(device): with self.device_module.device(device):
@@ -237,7 +237,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
self.to(torch.float32) self.to(torch.float32)
devices = [parameter.device] if parameter.device.type != "cpu" else [] devices = [parameter.device] if parameter.device.type != "cpu" else []
try: try:
with torch.random.fork_rng(devices=devices): with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED) torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
for device in devices: for device in devices:
with self.device_module.device(device): with self.device_module.device(device):