Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ac67e8874d | ||
|
|
dfe6e13cfc | ||
|
|
9d80efcbe5 | ||
|
|
ddbbfd29e5 |
No files matched your search
+41
-23
@@ -101,31 +101,49 @@ nav:
|
||||
- CI: contributing/ci
|
||||
- Design Documents:
|
||||
- design/index.md
|
||||
- design/architecture_overview.md
|
||||
- Architecture Documents:
|
||||
- Architecture Overview: design/architecture_overview.md
|
||||
- Feature Design:
|
||||
- design/feature/disaggregated_inference.md
|
||||
- design/feature/ray_based_execution.md
|
||||
- design/feature/omni_connectors/
|
||||
- design/feature/prefix_caching.md
|
||||
- design/feature/cfg_parallel.md
|
||||
- design/feature/expert_parallel.md
|
||||
- design/feature/sequence_parallel.md
|
||||
- design/feature/tensor_parallel.md
|
||||
- design/feature/vae_parallel.md
|
||||
- design/feature/hsdp.md
|
||||
- design/feature/cache_dit.md
|
||||
- design/feature/teacache.md
|
||||
- design/feature/skip_softmax.md
|
||||
- design/feature/async_chunk.md
|
||||
- design/feature/omni_async_output_materialization.md
|
||||
- design/feature/vae_parallel.md
|
||||
- design/feature/diffusion_continuous_batching.md
|
||||
- design/feature/distributed_layerwise_offload.md
|
||||
- Runtime and Stage Execution:
|
||||
- Disaggregated Inference: design/feature/disaggregated_inference.md
|
||||
- Ray-based Execution: design/feature/ray_based_execution.md
|
||||
- Async Chunk: design/feature/async_chunk.md
|
||||
- Async Diffusion Output: design/feature/async_diffusion_output.md
|
||||
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
|
||||
- Automatic Prefix Caching: design/feature/prefix_caching.md
|
||||
- Communication:
|
||||
- OmniConnector Implementations:
|
||||
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
|
||||
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
|
||||
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
|
||||
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||
- Diffusion Acceleration:
|
||||
- Parallelism:
|
||||
- CFG-Parallel: design/feature/cfg_parallel.md
|
||||
- Expert Parallel: design/feature/expert_parallel.md
|
||||
- Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md
|
||||
- Pipeline Parallel: design/feature/pipeline_parallel.md
|
||||
- Sequence Parallel: design/feature/sequence_parallel.md
|
||||
- Tensor Parallel: design/feature/tensor_parallel.md
|
||||
- VAE Patch Parallelism: design/feature/vae_parallel.md
|
||||
- Attention Backends:
|
||||
- Skip-Softmax: design/feature/skip_softmax.md
|
||||
- Quantization: user_guide/quantization/overview.md
|
||||
- Cache-DiT: design/feature/cache_dit.md
|
||||
- TeaCache: design/feature/teacache.md
|
||||
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||
- Infrastructure and Performance:
|
||||
- Prometheus Metrics: design/metrics.md
|
||||
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
|
||||
- Module Design:
|
||||
- design/module/ar_module.md
|
||||
- design/module/dit_module.md
|
||||
- design/module/entrypoint_module.md
|
||||
- design/module/async_omni_architecture.md
|
||||
- Runtime Modules:
|
||||
- AR Module: design/module/ar_module.md
|
||||
- DIT Module: design/module/dit_module.md
|
||||
- Orchestration:
|
||||
- Entrypoint Module: design/module/entrypoint_module.md
|
||||
- AsyncOmni Architecture: design/module/async_omni_architecture.md
|
||||
- Docs Guide: contributing/DOCS_GUIDE.md
|
||||
- API Reference:
|
||||
- api/README.md
|
||||
|
||||
@@ -1,156 +0,0 @@
|
||||
# Distributed Layerwise Offload
|
||||
|
||||
This document describes distributed layerwise offload (DLO) for diffusion
|
||||
models. DLO keeps only a small number of DiT blocks on the accelerator and
|
||||
streams the remaining blocks from host memory. The distributed backend can
|
||||
either shard those host-side weights across an existing parallel group or keep
|
||||
the standard loader's rank-local weights and avoid an additional collective.
|
||||
|
||||
For user-facing commands, see the
|
||||
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload_diffusion.md)
|
||||
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
|
||||
|
||||
## Status
|
||||
|
||||
DLO is implemented for multi-device diffusion execution. The default
|
||||
AllGather path is the primary path for DP and SP deployments. The
|
||||
`--dlo-no-use-allgather` path is a rank-local compatibility mode: it is useful
|
||||
for standard-loader sharding, workstation bring-up, and systems where an
|
||||
additional DLO collective is undesirable, but it does not reduce host weight
|
||||
storage across ranks.
|
||||
|
||||
The compatibility matrix below describes the current implementation. The
|
||||
unit-level guards are covered, but not every parallelism combination has a
|
||||
full model-and-hardware end-to-end test.
|
||||
|
||||
## Design
|
||||
|
||||
### DLO consumes the existing parallel topology
|
||||
|
||||
DLO does not create a new DP, TP, or SP topology. It reads the configured
|
||||
`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
|
||||
standard distributed groups have been initialized.
|
||||
|
||||
The DLO weight-sharding group is selected as follows:
|
||||
|
||||
1. Use the existing DP group when `data_parallel_size > 1`.
|
||||
2. When DP is one and SP is greater than one, use the SP group.
|
||||
3. Otherwise, run rank-locally without a DLO process group.
|
||||
|
||||
TP is deliberately not used as DLO's AllGather group. HSDP has its own
|
||||
parameter-sharding lifecycle and is not allowed to be sharded a second time by
|
||||
DLO's AllGather path.
|
||||
|
||||
### AllGather path
|
||||
|
||||
With the default `dlo_use_allgather=True`, each rank stores approximately
|
||||
`1 / group_size` of each streamable block in pinned host memory. The next
|
||||
block's shard is copied to a device buffer and reconstructed with
|
||||
`all_gather_into_tensor` on a communication stream while the current block is
|
||||
executing.
|
||||
|
||||
```text
|
||||
Compute: [Block N] [Block N+1] [Block N+2]
|
||||
H2D: [shard N+1] [shard N+2]
|
||||
AllGather: [full N+1] [full N+2]
|
||||
Buffers: [current slot] [prefetch slot] [current slot]
|
||||
```
|
||||
|
||||

|
||||
|
||||
The backend uses two shared device buffers, so accelerator weight residency is
|
||||
bounded by the largest streamed blocks rather than the complete model.
|
||||
|
||||
When DP is greater than one, the engine can process one request per DP rank in
|
||||
the same denoising wave. Because AllGather is a collective, all participating
|
||||
requests must take the same execution path at every denoising step.
|
||||
|
||||
### Rank-local path without DLO AllGather
|
||||
|
||||
With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
|
||||
one. The regular model loader remains responsible for preparing each rank's
|
||||
weights, including TP-local tensors or HSDP-managed parameters. DLO then
|
||||
streams those rank-local tensors block by block using H2D copies only.
|
||||
|
||||
This mode means:
|
||||
|
||||
- DP still provides independent replicas, but DLO does not shard weights
|
||||
across DP ranks.
|
||||
- SP still performs its normal activation/attention collectives, but DLO does
|
||||
not shard weights across SP ranks.
|
||||
- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
|
||||
DLO's additional weight AllGather is disabled.
|
||||
- Pure DP deployments keep a full host-side model copy per rank, subject to
|
||||
shared OS page-cache behavior.
|
||||
- The scheduler does not require a synchronized DP request wave for DLO.
|
||||
|
||||
This path is intentionally not implemented by reusing the AllGather mmap
|
||||
loader. It relies on the standard loader so model-specific TP/HSDP transforms
|
||||
remain intact.
|
||||
|
||||
## Parallelism compatibility
|
||||
|
||||
| Parallelism | DLO + AllGather | DLO without AllGather |
|
||||
|---|---|---|
|
||||
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. DP replicas remain independent; no DLO weight sharding or cross-DP DLO collective. |
|
||||
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
|
||||
| **TP > 1** | Unsupported in the DLO mmap path. TP-aware loader callbacks are bypassed, so the backend rejects this configuration when it enters that path. | Standard loading is retained and TP-local tensors can be streamed. This is the intended compatibility path, but it still needs broader model and hardware validation. |
|
||||
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
|
||||
|
||||
### Combined dimensions
|
||||
|
||||
- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
|
||||
than one; SP continues to use its own sequence-parallel group. If DP is one,
|
||||
the SP group becomes DLO's sharding group in AllGather mode.
|
||||
- **DP + TP/SP without AllGather:** standard model loading defines the
|
||||
rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
|
||||
collective.
|
||||
- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
|
||||
DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
|
||||
materialization and synchronization.
|
||||
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
|
||||
configuration.
|
||||
|
||||
## Request and loading constraints
|
||||
|
||||
AllGather DP multi-concurrency requires:
|
||||
|
||||
- explicit `num_inference_steps`;
|
||||
- the same `num_inference_steps` for all requests in a wave; and
|
||||
- identical request arguments that affect the collective execution path.
|
||||
|
||||
The no-AllGather path does not impose these DLO-specific synchronized-wave
|
||||
requirements.
|
||||
|
||||
The mmap loader is used only by the supported DLO+AllGather path when the
|
||||
model has a compatible checkpoint layout. Online quantization is incompatible
|
||||
with that sharded mmap path; use `--dlo-no-use-allgather` or disable online
|
||||
quantization. Models that do not meet the mmap requirements use the regular
|
||||
loader path.
|
||||
|
||||
## Validation coverage
|
||||
|
||||
Current source-level validation includes:
|
||||
|
||||
- HSDP + DLO + AllGather rejection;
|
||||
- HSDP + DLO without AllGather acceptance at configuration level;
|
||||
- TP rejection in the DLO+AllGather mmap path;
|
||||
- resident-layer requests requiring no-AllGather;
|
||||
- DP request-wave validation for denoising-step compatibility;
|
||||
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
|
||||
tests.
|
||||
|
||||
The highest-value missing coverage is end-to-end numerical comparison against
|
||||
ordinary layerwise offload for DP+SP, TP+no-AllGather, and HSDP+SP+no-AllGather
|
||||
on the target CUDA/NCCL or CANN/HCCL hardware.
|
||||
|
||||
## Recommendations
|
||||
|
||||
- Use **DP + DLO AllGather** for the supported throughput and host-memory
|
||||
scaling path.
|
||||
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
|
||||
not the goal.
|
||||
- Use **no-AllGather** to bring up TP or workstation PCIe configurations, with
|
||||
the expectation of higher host-memory use and lower validation confidence.
|
||||
- Prefer **HSDP alone** for production HSDP deployments until the combined
|
||||
HSDP + DLO no-AllGather path has broader end-to-end coverage.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 195 KiB |
+51
-4
@@ -1,6 +1,8 @@
|
||||
# Design Documents
|
||||
|
||||
This section contains design documents and architecture specifications for vLLM-Omni.
|
||||
This section contains design documents and architecture specifications for
|
||||
vLLM-Omni. The sidebar groups documents by the system concern they describe,
|
||||
while the source paths remain stable so existing links continue to work.
|
||||
|
||||
## Architecture Documents
|
||||
|
||||
@@ -8,18 +10,63 @@ This section contains design documents and architecture specifications for vLLM-
|
||||
|
||||
## Feature Design Documents
|
||||
|
||||
### Runtime and stage execution
|
||||
|
||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
- [Ray-based Execution](feature/ray_based_execution.md)
|
||||
- [Async Chunk](feature/async_chunk.md)
|
||||
- [Async Diffusion Output](feature/async_diffusion_output.md)
|
||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
|
||||
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
|
||||
|
||||
## Infrastructure Design Documents
|
||||
### Communication
|
||||
|
||||
#### OmniConnector implementations
|
||||
|
||||
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
|
||||
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
|
||||
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
|
||||
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
|
||||
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||
|
||||
### Diffusion acceleration
|
||||
|
||||
#### Parallelism
|
||||
|
||||
- [CFG-Parallel](feature/cfg_parallel.md)
|
||||
- [Expert Parallel](feature/expert_parallel.md)
|
||||
- [Hybrid Sharded Data Parallel (HSDP)](feature/hsdp.md)
|
||||
- [Pipeline Parallel](feature/pipeline_parallel.md)
|
||||
- [Sequence Parallel](feature/sequence_parallel.md)
|
||||
- [Tensor Parallel](feature/tensor_parallel.md)
|
||||
- [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||
|
||||
#### Attention Backends
|
||||
|
||||
- [Skip-Softmax](feature/skip_softmax.md)
|
||||
|
||||
#### Quantization
|
||||
|
||||
- [Quantization Overview](../user_guide/quantization/overview.md)
|
||||
|
||||
- [Cache-DiT](feature/cache_dit.md)
|
||||
- [TeaCache](feature/teacache.md)
|
||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||
|
||||
## Infrastructure and Performance
|
||||
|
||||
- [Prometheus Metrics](metrics.md)
|
||||
- [Speech Generation Performance Optimizations](qwen3_omni_tts_performance_optimization.md)
|
||||
|
||||
## Module Design Documents
|
||||
|
||||
### Runtime Modules
|
||||
|
||||
- [AR Module](module/ar_module.md)
|
||||
- [DIT Module](module/dit_module.md)
|
||||
|
||||
### Orchestration
|
||||
|
||||
- [Entrypoint Module](module/entrypoint_module.md)
|
||||
- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md)
|
||||
@@ -184,15 +184,10 @@ class Flux2Transformer2DModel(nn.Module):
|
||||
### How It Works
|
||||
|
||||
Distributed layerwise offloading extends single-GPU layerwise offloading to
|
||||
multi-device deployments. In the default mode, each rank stores only
|
||||
**1/dp_size** of the model weights in host memory; full layer weights are
|
||||
reconstructed at runtime via **AllGather** on a dedicated communication stream,
|
||||
overlapped with computation via a fixed double-buffer scheme. The
|
||||
`--dlo-no-use-allgather` mode instead streams standard-loader rank-local
|
||||
weights independently and does not shard weights across ranks.
|
||||
|
||||
For the implementation design and compatibility matrix, see
|
||||
[Distributed Layerwise Offload](../../design/feature/distributed_layerwise_offload.md).
|
||||
multi-device deployments. Each DP rank stores only **1/dp_size** of the model
|
||||
weights in host memory; full layer weights are reconstructed at runtime via
|
||||
**AllGather** on a dedicated communication stream, overlapped with computation
|
||||
via a fixed double-buffer scheme.
|
||||
|
||||
**Key features:**
|
||||
- **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
|
||||
@@ -224,7 +219,7 @@ vllm serve /path/to/model --omni \
|
||||
--enable-distributed-layerwise-offload \
|
||||
--data-parallel-size 4
|
||||
|
||||
# Without DLO AllGather (each rank streams standard-loader rank-local weights)
|
||||
# Without AllGather (each rank loads full weights, no sharding)
|
||||
vllm serve /path/to/model --omni \
|
||||
--enable-distributed-layerwise-offload \
|
||||
--data-parallel-size 4 \
|
||||
@@ -252,9 +247,9 @@ m = Omni(
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
|
||||
| `--data-parallel-size N` | Number of DP ranks; DLO weight-sharding group when AllGather is enabled | `1` |
|
||||
| `--data-parallel-size N` | Number of DP ranks for weight sharding | `1` |
|
||||
| `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
|
||||
| `--dlo-no-use-allgather` | Use standard-loader rank-local weights; no DLO sharding or AllGather | `false` |
|
||||
| `--dlo-no-use-allgather` | Full weights per rank (no sharding, no AllGather) | `false` |
|
||||
|
||||
### How model weights are loaded (mmap path)
|
||||
|
||||
@@ -308,15 +303,9 @@ request while AllGather synchronizes only weight shards (request-independent).
|
||||
|
||||
### Limitations
|
||||
- Online quantization (FP8) is incompatible with mmap loading — falls back to
|
||||
regular loading only when the model is outside the DLO+AllGather mmap path;
|
||||
online quantization with DLO+AllGather is rejected. Use
|
||||
`--dlo-no-use-allgather` or disable online quantization.
|
||||
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path because TP-aware
|
||||
weight-loader callbacks are bypassed. The no-AllGather path retains the
|
||||
standard TP loader and is experimental.
|
||||
- HSDP + AllGather is rejected because it would double-shard parameters. HSDP
|
||||
with no-AllGather is accepted at configuration level, but has limited
|
||||
end-to-end validation.
|
||||
regular `load_weights()` automatically
|
||||
- Tensor Parallel is not supported (DLO uses DP-based sharding)
|
||||
- HSDP + AllGather is rejected (would double-shard weights)
|
||||
- `num_inference_steps=None` is not allowed in DP multi-concurrency mode
|
||||
|
||||
**Module Discovery**
|
||||
@@ -349,7 +338,7 @@ OffloadBackend (base class)
|
||||
├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
|
||||
│ (single-GPU, full weights on host)
|
||||
└── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
|
||||
(multi-GPU, sharded AllGather or rank-local mode)
|
||||
(multi-GPU, 1/dp_size sharded weights + AllGather)
|
||||
```
|
||||
|
||||
Factory function `get_offload_backend()` selects the appropriate backend based on
|
||||
@@ -380,10 +369,4 @@ reasoner/generator components inside the model forward pass.
|
||||
**Notes:**
|
||||
- Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
|
||||
- Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
|
||||
- Distributed Layerwise Offloading reuses the layerwise block topology, but the
|
||||
model and parallelism combination still needs validation. AllGather mode
|
||||
additionally requires a compatible checkpoint/mmap layout; no-AllGather mode
|
||||
retains the standard loader's rank-local tensors. See the
|
||||
[DLO design](../../design/feature/distributed_layerwise_offload.md) and
|
||||
[Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
|
||||
for usage examples.
|
||||
- Distributed Layerwise Offloading works with any model that supports Layerwise Offloading — no additional model changes required. See [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md) for usage examples.
|
||||
@@ -56,14 +56,10 @@ vllm serve /path/to/Cosmos3-Super \
|
||||
--data-parallel-size 4
|
||||
```
|
||||
|
||||
### Without DLO AllGather (rank-local weights, no DLO sharding)
|
||||
### Without AllGather (full weights per rank, no sharding)
|
||||
|
||||
Use `--dlo-no-use-allgather` when you want each rank to stream the standard
|
||||
loader's rank-local weights independently — no DLO AllGather synchronization,
|
||||
no concurrent request requirement, but no DLO weight sharding and higher host
|
||||
memory use. In pure DP, this is equivalent to independent full-weight
|
||||
layerwise-offload replicas; with TP or HSDP, each rank retains the standard
|
||||
loader's local tensor layout.
|
||||
Use `--dlo-no-use-allgather` when you want each rank to load full weights independently —
|
||||
no AllGather synchronization, no concurrent request requirement, but N× host memory.
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/Cosmos3-Nano \
|
||||
@@ -158,9 +154,9 @@ wait
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` |
|
||||
| `--data-parallel-size N` | Number of DP ranks; DLO weight sharding + concurrent requests when AllGather is enabled | `1` |
|
||||
| `--data-parallel-size N` | Number of DP ranks (weight sharding + concurrent requests) | `1` |
|
||||
| `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` |
|
||||
| `--dlo-no-use-allgather` | Stream standard-loader rank-local weights independently (no DLO sharding, no DLO AllGather) | `false` |
|
||||
| `--dlo-no-use-allgather` | Each rank loads full weights independently (no sharding, no AllGather) | `false` |
|
||||
| `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` |
|
||||
|
||||
## Mode comparison
|
||||
@@ -170,7 +166,7 @@ wait
|
||||
| DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences |
|
||||
| DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory |
|
||||
| DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) |
|
||||
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Standard-loader rank-local weights | 2 blocks | 0.4× HSDP | No DLO AllGather overhead, high CPU |
|
||||
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Full model | 2 blocks | 0.4× HSDP | No AllGather overhead, high CPU |
|
||||
| HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM |
|
||||
|
||||
## Memory expectations
|
||||
@@ -189,11 +185,7 @@ wait
|
||||
|
||||
- `--dlo-use-allgather` (default) requires all concurrent requests to have the same
|
||||
`num_inference_steps` (AllGather is a collective that needs all ranks synchronized)
|
||||
- Online quantization (FP8) is incompatible with the DLO+AllGather mmap path;
|
||||
use `--dlo-no-use-allgather` or disable online quantization
|
||||
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path. TP with
|
||||
`--dlo-no-use-allgather` retains the standard TP loader and should be treated
|
||||
as an experimental compatibility path
|
||||
- HSDP + AllGather is rejected (would double-shard weights). HSDP with
|
||||
`--dlo-no-use-allgather` is accepted at configuration level, but is not yet a
|
||||
broadly validated production combination
|
||||
- Online quantization (FP8) is incompatible with mmap loading — falls back to regular
|
||||
`load_weights()` automatically
|
||||
- Tensor Parallel is not supported (DLO uses DP-based sharding, a different dimension)
|
||||
- HSDP + AllGather is rejected (would double-shard weights)
|
||||
@@ -2,9 +2,6 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Unit tests for LayerNorm and RMSNorm custom ops in diffusion layers."""
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from pytest_mock import MockerFixture
|
||||
@@ -302,44 +299,6 @@ def test_rmsnorm_numerical_correctness():
|
||||
torch.testing.assert_close(out, expected, atol=1e-5, rtol=1e-5)
|
||||
|
||||
|
||||
def test_rmsnorm_dtype_matches_parameter_and_fp32_accumulation_contract():
|
||||
"""dtype controls gamma storage; reduced-precision inputs still use fp32 accumulation."""
|
||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
||||
|
||||
hidden_size = 64
|
||||
eps = 1e-6
|
||||
norm = RMSNorm(hidden_size, eps=eps, dtype=torch.bfloat16)
|
||||
x = torch.randn(2, 4, hidden_size, dtype=torch.bfloat16)
|
||||
|
||||
assert norm.weight.dtype == torch.bfloat16
|
||||
|
||||
x_fp32 = x.float()
|
||||
expected = (x_fp32 * torch.rsqrt(x_fp32.square().mean(-1, keepdim=True) + eps) * norm.weight.float()).to(x.dtype)
|
||||
|
||||
torch.testing.assert_close(norm.forward_native(x), expected, atol=0, rtol=0)
|
||||
|
||||
|
||||
def test_rmsnorm_npu_receives_gamma_with_configured_dtype(monkeypatch: pytest.MonkeyPatch):
|
||||
"""The NPU fused op receives the bf16 checkpoint gamma used by H3."""
|
||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
||||
|
||||
captured: dict[str, torch.Tensor] = {}
|
||||
|
||||
def npu_rms_norm(x: torch.Tensor, gamma: torch.Tensor, epsilon: float):
|
||||
captured["gamma"] = gamma
|
||||
captured["epsilon"] = torch.tensor(epsilon)
|
||||
return (x,)
|
||||
|
||||
monkeypatch.setitem(sys.modules, "torch_npu", types.SimpleNamespace(npu_rms_norm=npu_rms_norm))
|
||||
norm = RMSNorm(64, eps=1e-5, dtype=torch.bfloat16)
|
||||
x = torch.randn(2, 4, 64, dtype=torch.bfloat16)
|
||||
|
||||
assert norm.forward_npu(x) is x
|
||||
assert captured["gamma"] is norm.weight
|
||||
assert captured["gamma"].dtype == torch.bfloat16
|
||||
assert captured["epsilon"].item() == pytest.approx(1e-5)
|
||||
|
||||
|
||||
# ── RMSNorm compile-path regression tests ──
|
||||
|
||||
|
||||
|
||||
@@ -10,13 +10,10 @@ forward_hip, and apply_rotary_emb_mindiesd paths.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding, apply_rotary_emb_mindiesd
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
@@ -237,33 +234,3 @@ class TestRotaryEmbeddingNativeShapeRegression:
|
||||
output = rope.forward_native(x, cos, sin)
|
||||
assert output.shape == x.shape
|
||||
assert not torch.isnan(output).any()
|
||||
|
||||
|
||||
def test_mindie_full_dim_rope_keeps_h3_cos_sin_layout(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""H3 passes complete rotary dims to MindIE instead of the CUDA half-dim layout."""
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def rotary_position_embedding(x, cos, sin, **kwargs):
|
||||
captured.update(x=x, cos=cos, sin=sin, kwargs=kwargs)
|
||||
return x
|
||||
|
||||
monkeypatch.setitem(
|
||||
sys.modules,
|
||||
"mindiesd",
|
||||
types.SimpleNamespace(rotary_position_embedding=rotary_position_embedding),
|
||||
)
|
||||
x = torch.randn(11, 3, 96, dtype=torch.bfloat16)
|
||||
cos = torch.randn(11, 96, dtype=torch.bfloat16)
|
||||
sin = torch.randn(11, 96, dtype=torch.bfloat16)
|
||||
|
||||
actual = apply_rotary_emb_mindiesd(x, cos, sin, interleaved=False, half_head_dim=False)
|
||||
|
||||
assert actual is x
|
||||
assert captured["x"] is x
|
||||
assert captured["cos"] is cos
|
||||
assert captured["sin"] is sin
|
||||
assert captured["kwargs"] == {
|
||||
"rotated_mode": "rotated_half",
|
||||
"head_first": False,
|
||||
"fused": True,
|
||||
}
|
||||
@@ -61,33 +61,6 @@ def test_packed_attention_is_a_regional_compile_boundary():
|
||||
assert getattr(MiniMaxH3Attention._run_packed_attention, "_torchdynamo_disable", False)
|
||||
|
||||
|
||||
def test_h3_fused_rope_matches_reference_and_preserves_unrotated_dims():
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
||||
from vllm_omni.diffusion.models.minimax_h3.minimax_h3_transformer import (
|
||||
MiniMaxH3Attention,
|
||||
)
|
||||
|
||||
attention = object.__new__(MiniMaxH3Attention)
|
||||
nn.Module.__init__(attention)
|
||||
attention.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
|
||||
attention.rope._forward_method = attention.rope.forward_native
|
||||
|
||||
x = torch.randn(11, 3, 128, dtype=torch.bfloat16)
|
||||
freqs_half = torch.randn(11, 48)
|
||||
freqs = torch.cat((freqs_half, freqs_half), dim=-1)
|
||||
actual = attention._apply_rope(x, freqs)
|
||||
|
||||
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1)
|
||||
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
|
||||
x_rot = x[..., :96]
|
||||
x1, x2 = x_rot.chunk(2, dim=-1)
|
||||
expected_rot = x_rot * cos + torch.cat((-x2, x1), dim=-1) * sin
|
||||
expected = torch.cat((expected_rot, x[..., 96:]), dim=-1)
|
||||
|
||||
torch.testing.assert_close(actual, expected, atol=0, rtol=0)
|
||||
torch.testing.assert_close(actual[..., 96:], x[..., 96:], atol=0, rtol=0)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("tp_size", "message"),
|
||||
[
|
||||
|
||||
@@ -135,7 +135,6 @@ def _generate_joint_output(omni, config: QualityTestConfig):
|
||||
num_frames=config.num_frames,
|
||||
guidance_scale=config.guidance_scale,
|
||||
sigmas=config.sigmas,
|
||||
extra_args={"aspect_ratio": "16:9"},
|
||||
),
|
||||
)
|
||||
main_process_peak_mem = torch.accelerator.max_memory_allocated() / (1024**3)
|
||||
|
||||
@@ -65,9 +65,9 @@ class LayerNorm(nn.LayerNorm, CustomOp):
|
||||
|
||||
|
||||
class RMSNorm(CustomOp):
|
||||
def __init__(self, hidden_size: int, eps: float = 1e-6, dtype: torch.dtype = torch.float32) -> None:
|
||||
def __init__(self, hidden_size: int, eps: float = 1e-6) -> None:
|
||||
super().__init__()
|
||||
self.weight = nn.Parameter(torch.ones(hidden_size, dtype=dtype))
|
||||
self.weight = nn.Parameter(torch.ones(hidden_size))
|
||||
self.variance_epsilon = eps
|
||||
|
||||
def _forward_fused(self, x: torch.Tensor) -> torch.Tensor:
|
||||
|
||||
@@ -89,11 +89,10 @@ class RotaryEmbedding(CustomOp):
|
||||
of 1st half and 2nd half (GPT-NeoX style).
|
||||
"""
|
||||
|
||||
def __init__(self, is_neox_style: bool = False, half_head_dim: bool = True) -> None:
|
||||
def __init__(self, is_neox_style: bool = False) -> None:
|
||||
super().__init__()
|
||||
self.is_neox_style = is_neox_style
|
||||
self.interleaved = not is_neox_style
|
||||
self.half_head_dim = half_head_dim
|
||||
self.apply_rotary_emb_flash_attn = None
|
||||
self.has_mindie = False
|
||||
# ``find_spec("flash_attn")`` is True as long as *any* package publishes
|
||||
@@ -111,21 +110,6 @@ class RotaryEmbedding(CustomOp):
|
||||
if find_spec("mindiesd") is not None:
|
||||
self.has_mindie = True
|
||||
|
||||
def _prepare_half_head_dim_cos_sin(
|
||||
self,
|
||||
cos: torch.Tensor,
|
||||
sin: torch.Tensor,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Normalize shared cos/sin to the half-dim layout used by CUDA/native kernels."""
|
||||
if cos.dim() == 3:
|
||||
# All batch elements share the same rotary position encoding.
|
||||
cos = cos[0]
|
||||
sin = sin[0]
|
||||
if not self.half_head_dim:
|
||||
cos = cos[..., ::2] if self.interleaved else cos[..., : cos.shape[-1] // 2]
|
||||
sin = sin[..., ::2] if self.interleaved else sin[..., : sin.shape[-1] // 2]
|
||||
return cos, sin
|
||||
|
||||
def forward_cuda(
|
||||
self,
|
||||
x: torch.Tensor,
|
||||
@@ -134,7 +118,10 @@ class RotaryEmbedding(CustomOp):
|
||||
) -> torch.Tensor:
|
||||
from vllm.vllm_flash_attn.layers.rotary import apply_rotary_emb
|
||||
|
||||
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
|
||||
if cos.dim() == 3:
|
||||
# (B, S, D/2) -> (S, D/2)
|
||||
cos = cos[0]
|
||||
sin = sin[0]
|
||||
|
||||
x, squeezed = _ensure_batch_dim(x)
|
||||
output = apply_rotary_emb(
|
||||
@@ -154,7 +141,10 @@ class RotaryEmbedding(CustomOp):
|
||||
if self.apply_rotary_emb_flash_attn is None:
|
||||
return self.forward_cuda(x, cos, sin)
|
||||
|
||||
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
|
||||
if cos.dim() == 3:
|
||||
# (B, S, D/2) -> (S, D/2)
|
||||
cos = cos[0]
|
||||
sin = sin[0]
|
||||
|
||||
x, squeezed = _ensure_batch_dim(x)
|
||||
output = self.apply_rotary_emb_flash_attn(
|
||||
@@ -172,7 +162,7 @@ class RotaryEmbedding(CustomOp):
|
||||
sin: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
if self.has_mindie:
|
||||
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved, self.half_head_dim)
|
||||
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved)
|
||||
else:
|
||||
return self.forward_native(x, cos, sin)
|
||||
|
||||
@@ -201,7 +191,9 @@ class RotaryEmbedding(CustomOp):
|
||||
# All batch elements share the same rotary position encoding.
|
||||
# Strip the batch dim so the underlying op broadcasts over the batch,
|
||||
# consistent with forward_cuda / forward_hip / apply_rotary_emb_mindiesd.
|
||||
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
|
||||
if cos.dim() == 3:
|
||||
cos = cos[0]
|
||||
sin = sin[0]
|
||||
return apply_rotary_emb_torch(
|
||||
x,
|
||||
cos,
|
||||
|
||||
@@ -33,8 +33,6 @@ from vllm_omni.diffusion.distributed.sp_plan import (
|
||||
SequenceParallelInput,
|
||||
SequenceParallelOutput,
|
||||
)
|
||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.model_executor.layers.quantization.base_config import (
|
||||
@@ -170,11 +168,16 @@ def _reorder_grouped_qkv_to_qkv(
|
||||
)
|
||||
|
||||
|
||||
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> RMSNorm:
|
||||
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> nn.RMSNorm:
|
||||
# RMSNorm uses fp32 accumulation with bf16 inputs and outputs.
|
||||
# torch.nn.RMSNorm upcasts reduced-precision inputs for the variance
|
||||
# reduction, matching that accumulation semantic.
|
||||
return RMSNorm(size, eps=eps, dtype=dtype)
|
||||
return nn.RMSNorm(size, eps=eps, dtype=dtype)
|
||||
|
||||
|
||||
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
|
||||
x1, x2 = torch.chunk(x, 2, dim=-1)
|
||||
return torch.cat((-x2, x1), dim=-1)
|
||||
|
||||
|
||||
def _modulate_scale_shift(
|
||||
@@ -227,6 +230,20 @@ class MiniMaxH3Rope(nn.Module):
|
||||
return torch.cat((half, half), dim=-1) # [S, 96]
|
||||
|
||||
|
||||
def _apply_rope(x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
|
||||
"""Rotate the first rot_dim head dims; pass the rest through.
|
||||
|
||||
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
|
||||
are cast to the activation dtype before the elementwise math.
|
||||
"""
|
||||
rot_dim = freqs.shape[-1]
|
||||
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
|
||||
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1) # [T, 1, rot_dim]
|
||||
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
|
||||
x_rot = (x_rot * cos) + (_rotate_half(x_rot) * sin)
|
||||
return torch.cat((x_rot, x_pass), dim=-1)
|
||||
|
||||
|
||||
class MiniMaxH3TimeEmbedder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
@@ -332,7 +349,6 @@ class MiniMaxH3Attention(nn.Module):
|
||||
self.num_kv_heads = self.qkv_proj.num_kv_heads
|
||||
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
||||
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
||||
self.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
|
||||
self.out_proj = RowParallelLinear(
|
||||
inner_dim,
|
||||
arch.hidden_size,
|
||||
@@ -351,19 +367,6 @@ class MiniMaxH3Attention(nn.Module):
|
||||
skip_sequence_parallel=skip_sequence_parallel,
|
||||
)
|
||||
|
||||
def _apply_rope(self, x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
|
||||
"""Rotate the first rot_dim head dims; pass the rest through.
|
||||
|
||||
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
|
||||
are cast to the activation dtype before the elementwise math.
|
||||
"""
|
||||
rot_dim = freqs.shape[-1]
|
||||
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
|
||||
cos = torch.cos(freqs).to(x.dtype) # [T, rot_dim]
|
||||
sin = torch.sin(freqs).to(x.dtype)
|
||||
x_rot = self.rope(x_rot, cos, sin)
|
||||
return torch.cat((x_rot, x_pass), dim=-1)
|
||||
|
||||
@torch.compiler.disable
|
||||
def _run_packed_attention(
|
||||
self,
|
||||
@@ -447,8 +450,8 @@ class MiniMaxH3Attention(nn.Module):
|
||||
q = self.q_norm(q)
|
||||
k = self.k_norm(k)
|
||||
if rope_freqs is not None:
|
||||
q = self._apply_rope(q, rope_freqs)
|
||||
k = self._apply_rope(k, rope_freqs)
|
||||
q = _apply_rope(q, rope_freqs)
|
||||
k = _apply_rope(k, rope_freqs)
|
||||
|
||||
# The packed layout uses a second document for alignment padding.
|
||||
# Local/Ulysses backends unpad it, while Ring keeps aligned rows for
|
||||
|
||||
@@ -193,7 +193,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
|
||||
self.to(torch.float32)
|
||||
devices = [parameter.device] if parameter.device.type != "cpu" else []
|
||||
try:
|
||||
with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
|
||||
with torch.random.fork_rng(devices=devices):
|
||||
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
|
||||
for device in devices:
|
||||
with self.device_module.device(device):
|
||||
@@ -237,7 +237,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
|
||||
self.to(torch.float32)
|
||||
devices = [parameter.device] if parameter.device.type != "cpu" else []
|
||||
try:
|
||||
with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
|
||||
with torch.random.fork_rng(devices=devices):
|
||||
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
|
||||
for device in devices:
|
||||
with self.device_module.device(device):
|
||||
|
||||
Reference in new issue
Block a user