Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ac67e8874d | ||
|
|
dfe6e13cfc | ||
|
|
9d80efcbe5 | ||
|
|
ddbbfd29e5 |
No files matched your search
+41
-23
@@ -101,31 +101,49 @@ nav:
|
|||||||
- CI: contributing/ci
|
- CI: contributing/ci
|
||||||
- Design Documents:
|
- Design Documents:
|
||||||
- design/index.md
|
- design/index.md
|
||||||
- design/architecture_overview.md
|
- Architecture Documents:
|
||||||
|
- Architecture Overview: design/architecture_overview.md
|
||||||
- Feature Design:
|
- Feature Design:
|
||||||
- design/feature/disaggregated_inference.md
|
- Runtime and Stage Execution:
|
||||||
- design/feature/ray_based_execution.md
|
- Disaggregated Inference: design/feature/disaggregated_inference.md
|
||||||
- design/feature/omni_connectors/
|
- Ray-based Execution: design/feature/ray_based_execution.md
|
||||||
- design/feature/prefix_caching.md
|
- Async Chunk: design/feature/async_chunk.md
|
||||||
- design/feature/cfg_parallel.md
|
- Async Diffusion Output: design/feature/async_diffusion_output.md
|
||||||
- design/feature/expert_parallel.md
|
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
|
||||||
- design/feature/sequence_parallel.md
|
- Automatic Prefix Caching: design/feature/prefix_caching.md
|
||||||
- design/feature/tensor_parallel.md
|
- Communication:
|
||||||
- design/feature/vae_parallel.md
|
- OmniConnector Implementations:
|
||||||
- design/feature/hsdp.md
|
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
|
||||||
- design/feature/cache_dit.md
|
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
|
||||||
- design/feature/teacache.md
|
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
|
||||||
- design/feature/skip_softmax.md
|
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||||
- design/feature/async_chunk.md
|
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||||
- design/feature/omni_async_output_materialization.md
|
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||||
- design/feature/vae_parallel.md
|
- Diffusion Acceleration:
|
||||||
- design/feature/diffusion_continuous_batching.md
|
- Parallelism:
|
||||||
- design/feature/distributed_layerwise_offload.md
|
- CFG-Parallel: design/feature/cfg_parallel.md
|
||||||
|
- Expert Parallel: design/feature/expert_parallel.md
|
||||||
|
- Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md
|
||||||
|
- Pipeline Parallel: design/feature/pipeline_parallel.md
|
||||||
|
- Sequence Parallel: design/feature/sequence_parallel.md
|
||||||
|
- Tensor Parallel: design/feature/tensor_parallel.md
|
||||||
|
- VAE Patch Parallelism: design/feature/vae_parallel.md
|
||||||
|
- Attention Backends:
|
||||||
|
- Skip-Softmax: design/feature/skip_softmax.md
|
||||||
|
- Quantization: user_guide/quantization/overview.md
|
||||||
|
- Cache-DiT: design/feature/cache_dit.md
|
||||||
|
- TeaCache: design/feature/teacache.md
|
||||||
|
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||||
|
- Infrastructure and Performance:
|
||||||
|
- Prometheus Metrics: design/metrics.md
|
||||||
|
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
|
||||||
- Module Design:
|
- Module Design:
|
||||||
- design/module/ar_module.md
|
- Runtime Modules:
|
||||||
- design/module/dit_module.md
|
- AR Module: design/module/ar_module.md
|
||||||
- design/module/entrypoint_module.md
|
- DIT Module: design/module/dit_module.md
|
||||||
- design/module/async_omni_architecture.md
|
- Orchestration:
|
||||||
|
- Entrypoint Module: design/module/entrypoint_module.md
|
||||||
|
- AsyncOmni Architecture: design/module/async_omni_architecture.md
|
||||||
- Docs Guide: contributing/DOCS_GUIDE.md
|
- Docs Guide: contributing/DOCS_GUIDE.md
|
||||||
- API Reference:
|
- API Reference:
|
||||||
- api/README.md
|
- api/README.md
|
||||||
|
|||||||
@@ -1,156 +0,0 @@
|
|||||||
# Distributed Layerwise Offload
|
|
||||||
|
|
||||||
This document describes distributed layerwise offload (DLO) for diffusion
|
|
||||||
models. DLO keeps only a small number of DiT blocks on the accelerator and
|
|
||||||
streams the remaining blocks from host memory. The distributed backend can
|
|
||||||
either shard those host-side weights across an existing parallel group or keep
|
|
||||||
the standard loader's rank-local weights and avoid an additional collective.
|
|
||||||
|
|
||||||
For user-facing commands, see the
|
|
||||||
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload_diffusion.md)
|
|
||||||
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
DLO is implemented for multi-device diffusion execution. The default
|
|
||||||
AllGather path is the primary path for DP and SP deployments. The
|
|
||||||
`--dlo-no-use-allgather` path is a rank-local compatibility mode: it is useful
|
|
||||||
for standard-loader sharding, workstation bring-up, and systems where an
|
|
||||||
additional DLO collective is undesirable, but it does not reduce host weight
|
|
||||||
storage across ranks.
|
|
||||||
|
|
||||||
The compatibility matrix below describes the current implementation. The
|
|
||||||
unit-level guards are covered, but not every parallelism combination has a
|
|
||||||
full model-and-hardware end-to-end test.
|
|
||||||
|
|
||||||
## Design
|
|
||||||
|
|
||||||
### DLO consumes the existing parallel topology
|
|
||||||
|
|
||||||
DLO does not create a new DP, TP, or SP topology. It reads the configured
|
|
||||||
`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
|
|
||||||
standard distributed groups have been initialized.
|
|
||||||
|
|
||||||
The DLO weight-sharding group is selected as follows:
|
|
||||||
|
|
||||||
1. Use the existing DP group when `data_parallel_size > 1`.
|
|
||||||
2. When DP is one and SP is greater than one, use the SP group.
|
|
||||||
3. Otherwise, run rank-locally without a DLO process group.
|
|
||||||
|
|
||||||
TP is deliberately not used as DLO's AllGather group. HSDP has its own
|
|
||||||
parameter-sharding lifecycle and is not allowed to be sharded a second time by
|
|
||||||
DLO's AllGather path.
|
|
||||||
|
|
||||||
### AllGather path
|
|
||||||
|
|
||||||
With the default `dlo_use_allgather=True`, each rank stores approximately
|
|
||||||
`1 / group_size` of each streamable block in pinned host memory. The next
|
|
||||||
block's shard is copied to a device buffer and reconstructed with
|
|
||||||
`all_gather_into_tensor` on a communication stream while the current block is
|
|
||||||
executing.
|
|
||||||
|
|
||||||
```text
|
|
||||||
Compute: [Block N] [Block N+1] [Block N+2]
|
|
||||||
H2D: [shard N+1] [shard N+2]
|
|
||||||
AllGather: [full N+1] [full N+2]
|
|
||||||
Buffers: [current slot] [prefetch slot] [current slot]
|
|
||||||
```
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
The backend uses two shared device buffers, so accelerator weight residency is
|
|
||||||
bounded by the largest streamed blocks rather than the complete model.
|
|
||||||
|
|
||||||
When DP is greater than one, the engine can process one request per DP rank in
|
|
||||||
the same denoising wave. Because AllGather is a collective, all participating
|
|
||||||
requests must take the same execution path at every denoising step.
|
|
||||||
|
|
||||||
### Rank-local path without DLO AllGather
|
|
||||||
|
|
||||||
With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
|
|
||||||
one. The regular model loader remains responsible for preparing each rank's
|
|
||||||
weights, including TP-local tensors or HSDP-managed parameters. DLO then
|
|
||||||
streams those rank-local tensors block by block using H2D copies only.
|
|
||||||
|
|
||||||
This mode means:
|
|
||||||
|
|
||||||
- DP still provides independent replicas, but DLO does not shard weights
|
|
||||||
across DP ranks.
|
|
||||||
- SP still performs its normal activation/attention collectives, but DLO does
|
|
||||||
not shard weights across SP ranks.
|
|
||||||
- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
|
|
||||||
DLO's additional weight AllGather is disabled.
|
|
||||||
- Pure DP deployments keep a full host-side model copy per rank, subject to
|
|
||||||
shared OS page-cache behavior.
|
|
||||||
- The scheduler does not require a synchronized DP request wave for DLO.
|
|
||||||
|
|
||||||
This path is intentionally not implemented by reusing the AllGather mmap
|
|
||||||
loader. It relies on the standard loader so model-specific TP/HSDP transforms
|
|
||||||
remain intact.
|
|
||||||
|
|
||||||
## Parallelism compatibility
|
|
||||||
|
|
||||||
| Parallelism | DLO + AllGather | DLO without AllGather |
|
|
||||||
|---|---|---|
|
|
||||||
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. DP replicas remain independent; no DLO weight sharding or cross-DP DLO collective. |
|
|
||||||
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
|
|
||||||
| **TP > 1** | Unsupported in the DLO mmap path. TP-aware loader callbacks are bypassed, so the backend rejects this configuration when it enters that path. | Standard loading is retained and TP-local tensors can be streamed. This is the intended compatibility path, but it still needs broader model and hardware validation. |
|
|
||||||
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
|
|
||||||
|
|
||||||
### Combined dimensions
|
|
||||||
|
|
||||||
- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
|
|
||||||
than one; SP continues to use its own sequence-parallel group. If DP is one,
|
|
||||||
the SP group becomes DLO's sharding group in AllGather mode.
|
|
||||||
- **DP + TP/SP without AllGather:** standard model loading defines the
|
|
||||||
rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
|
|
||||||
collective.
|
|
||||||
- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
|
|
||||||
DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
|
|
||||||
materialization and synchronization.
|
|
||||||
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
|
|
||||||
configuration.
|
|
||||||
|
|
||||||
## Request and loading constraints
|
|
||||||
|
|
||||||
AllGather DP multi-concurrency requires:
|
|
||||||
|
|
||||||
- explicit `num_inference_steps`;
|
|
||||||
- the same `num_inference_steps` for all requests in a wave; and
|
|
||||||
- identical request arguments that affect the collective execution path.
|
|
||||||
|
|
||||||
The no-AllGather path does not impose these DLO-specific synchronized-wave
|
|
||||||
requirements.
|
|
||||||
|
|
||||||
The mmap loader is used only by the supported DLO+AllGather path when the
|
|
||||||
model has a compatible checkpoint layout. Online quantization is incompatible
|
|
||||||
with that sharded mmap path; use `--dlo-no-use-allgather` or disable online
|
|
||||||
quantization. Models that do not meet the mmap requirements use the regular
|
|
||||||
loader path.
|
|
||||||
|
|
||||||
## Validation coverage
|
|
||||||
|
|
||||||
Current source-level validation includes:
|
|
||||||
|
|
||||||
- HSDP + DLO + AllGather rejection;
|
|
||||||
- HSDP + DLO without AllGather acceptance at configuration level;
|
|
||||||
- TP rejection in the DLO+AllGather mmap path;
|
|
||||||
- resident-layer requests requiring no-AllGather;
|
|
||||||
- DP request-wave validation for denoising-step compatibility;
|
|
||||||
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
|
|
||||||
tests.
|
|
||||||
|
|
||||||
The highest-value missing coverage is end-to-end numerical comparison against
|
|
||||||
ordinary layerwise offload for DP+SP, TP+no-AllGather, and HSDP+SP+no-AllGather
|
|
||||||
on the target CUDA/NCCL or CANN/HCCL hardware.
|
|
||||||
|
|
||||||
## Recommendations
|
|
||||||
|
|
||||||
- Use **DP + DLO AllGather** for the supported throughput and host-memory
|
|
||||||
scaling path.
|
|
||||||
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
|
|
||||||
not the goal.
|
|
||||||
- Use **no-AllGather** to bring up TP or workstation PCIe configurations, with
|
|
||||||
the expectation of higher host-memory use and lower validation confidence.
|
|
||||||
- Prefer **HSDP alone** for production HSDP deployments until the combined
|
|
||||||
HSDP + DLO no-AllGather path has broader end-to-end coverage.
|
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 195 KiB |
+51
-4
@@ -1,6 +1,8 @@
|
|||||||
# Design Documents
|
# Design Documents
|
||||||
|
|
||||||
This section contains design documents and architecture specifications for vLLM-Omni.
|
This section contains design documents and architecture specifications for
|
||||||
|
vLLM-Omni. The sidebar groups documents by the system concern they describe,
|
||||||
|
while the source paths remain stable so existing links continue to work.
|
||||||
|
|
||||||
## Architecture Documents
|
## Architecture Documents
|
||||||
|
|
||||||
@@ -8,18 +10,63 @@ This section contains design documents and architecture specifications for vLLM-
|
|||||||
|
|
||||||
## Feature Design Documents
|
## Feature Design Documents
|
||||||
|
|
||||||
|
### Runtime and stage execution
|
||||||
|
|
||||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||||
- [Ray-based Execution](feature/ray_based_execution.md)
|
- [Ray-based Execution](feature/ray_based_execution.md)
|
||||||
|
- [Async Chunk](feature/async_chunk.md)
|
||||||
|
- [Async Diffusion Output](feature/async_diffusion_output.md)
|
||||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
|
||||||
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
|
|
||||||
|
|
||||||
## Infrastructure Design Documents
|
### Communication
|
||||||
|
|
||||||
|
#### OmniConnector implementations
|
||||||
|
|
||||||
|
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
|
||||||
|
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
|
||||||
|
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
|
||||||
|
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
|
||||||
|
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||||
|
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||||
|
|
||||||
|
### Diffusion acceleration
|
||||||
|
|
||||||
|
#### Parallelism
|
||||||
|
|
||||||
|
- [CFG-Parallel](feature/cfg_parallel.md)
|
||||||
|
- [Expert Parallel](feature/expert_parallel.md)
|
||||||
|
- [Hybrid Sharded Data Parallel (HSDP)](feature/hsdp.md)
|
||||||
|
- [Pipeline Parallel](feature/pipeline_parallel.md)
|
||||||
|
- [Sequence Parallel](feature/sequence_parallel.md)
|
||||||
|
- [Tensor Parallel](feature/tensor_parallel.md)
|
||||||
|
- [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||||
|
|
||||||
|
#### Attention Backends
|
||||||
|
|
||||||
|
- [Skip-Softmax](feature/skip_softmax.md)
|
||||||
|
|
||||||
|
#### Quantization
|
||||||
|
|
||||||
|
- [Quantization Overview](../user_guide/quantization/overview.md)
|
||||||
|
|
||||||
|
- [Cache-DiT](feature/cache_dit.md)
|
||||||
|
- [TeaCache](feature/teacache.md)
|
||||||
|
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||||
|
|
||||||
|
## Infrastructure and Performance
|
||||||
|
|
||||||
- [Prometheus Metrics](metrics.md)
|
- [Prometheus Metrics](metrics.md)
|
||||||
|
- [Speech Generation Performance Optimizations](qwen3_omni_tts_performance_optimization.md)
|
||||||
|
|
||||||
## Module Design Documents
|
## Module Design Documents
|
||||||
|
|
||||||
|
### Runtime Modules
|
||||||
|
|
||||||
- [AR Module](module/ar_module.md)
|
- [AR Module](module/ar_module.md)
|
||||||
- [DIT Module](module/dit_module.md)
|
- [DIT Module](module/dit_module.md)
|
||||||
|
|
||||||
|
### Orchestration
|
||||||
|
|
||||||
- [Entrypoint Module](module/entrypoint_module.md)
|
- [Entrypoint Module](module/entrypoint_module.md)
|
||||||
|
- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md)
|
||||||
@@ -184,15 +184,10 @@ class Flux2Transformer2DModel(nn.Module):
|
|||||||
### How It Works
|
### How It Works
|
||||||
|
|
||||||
Distributed layerwise offloading extends single-GPU layerwise offloading to
|
Distributed layerwise offloading extends single-GPU layerwise offloading to
|
||||||
multi-device deployments. In the default mode, each rank stores only
|
multi-device deployments. Each DP rank stores only **1/dp_size** of the model
|
||||||
**1/dp_size** of the model weights in host memory; full layer weights are
|
weights in host memory; full layer weights are reconstructed at runtime via
|
||||||
reconstructed at runtime via **AllGather** on a dedicated communication stream,
|
**AllGather** on a dedicated communication stream, overlapped with computation
|
||||||
overlapped with computation via a fixed double-buffer scheme. The
|
via a fixed double-buffer scheme.
|
||||||
`--dlo-no-use-allgather` mode instead streams standard-loader rank-local
|
|
||||||
weights independently and does not shard weights across ranks.
|
|
||||||
|
|
||||||
For the implementation design and compatibility matrix, see
|
|
||||||
[Distributed Layerwise Offload](../../design/feature/distributed_layerwise_offload.md).
|
|
||||||
|
|
||||||
**Key features:**
|
**Key features:**
|
||||||
- **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
|
- **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
|
||||||
@@ -224,7 +219,7 @@ vllm serve /path/to/model --omni \
|
|||||||
--enable-distributed-layerwise-offload \
|
--enable-distributed-layerwise-offload \
|
||||||
--data-parallel-size 4
|
--data-parallel-size 4
|
||||||
|
|
||||||
# Without DLO AllGather (each rank streams standard-loader rank-local weights)
|
# Without AllGather (each rank loads full weights, no sharding)
|
||||||
vllm serve /path/to/model --omni \
|
vllm serve /path/to/model --omni \
|
||||||
--enable-distributed-layerwise-offload \
|
--enable-distributed-layerwise-offload \
|
||||||
--data-parallel-size 4 \
|
--data-parallel-size 4 \
|
||||||
@@ -252,9 +247,9 @@ m = Omni(
|
|||||||
| Flag | Description | Default |
|
| Flag | Description | Default |
|
||||||
|------|-------------|---------|
|
|------|-------------|---------|
|
||||||
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
|
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
|
||||||
| `--data-parallel-size N` | Number of DP ranks; DLO weight-sharding group when AllGather is enabled | `1` |
|
| `--data-parallel-size N` | Number of DP ranks for weight sharding | `1` |
|
||||||
| `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
|
| `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
|
||||||
| `--dlo-no-use-allgather` | Use standard-loader rank-local weights; no DLO sharding or AllGather | `false` |
|
| `--dlo-no-use-allgather` | Full weights per rank (no sharding, no AllGather) | `false` |
|
||||||
|
|
||||||
### How model weights are loaded (mmap path)
|
### How model weights are loaded (mmap path)
|
||||||
|
|
||||||
@@ -308,15 +303,9 @@ request while AllGather synchronizes only weight shards (request-independent).
|
|||||||
|
|
||||||
### Limitations
|
### Limitations
|
||||||
- Online quantization (FP8) is incompatible with mmap loading — falls back to
|
- Online quantization (FP8) is incompatible with mmap loading — falls back to
|
||||||
regular loading only when the model is outside the DLO+AllGather mmap path;
|
regular `load_weights()` automatically
|
||||||
online quantization with DLO+AllGather is rejected. Use
|
- Tensor Parallel is not supported (DLO uses DP-based sharding)
|
||||||
`--dlo-no-use-allgather` or disable online quantization.
|
- HSDP + AllGather is rejected (would double-shard weights)
|
||||||
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path because TP-aware
|
|
||||||
weight-loader callbacks are bypassed. The no-AllGather path retains the
|
|
||||||
standard TP loader and is experimental.
|
|
||||||
- HSDP + AllGather is rejected because it would double-shard parameters. HSDP
|
|
||||||
with no-AllGather is accepted at configuration level, but has limited
|
|
||||||
end-to-end validation.
|
|
||||||
- `num_inference_steps=None` is not allowed in DP multi-concurrency mode
|
- `num_inference_steps=None` is not allowed in DP multi-concurrency mode
|
||||||
|
|
||||||
**Module Discovery**
|
**Module Discovery**
|
||||||
@@ -349,7 +338,7 @@ OffloadBackend (base class)
|
|||||||
├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
|
├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
|
||||||
│ (single-GPU, full weights on host)
|
│ (single-GPU, full weights on host)
|
||||||
└── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
|
└── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
|
||||||
(multi-GPU, sharded AllGather or rank-local mode)
|
(multi-GPU, 1/dp_size sharded weights + AllGather)
|
||||||
```
|
```
|
||||||
|
|
||||||
Factory function `get_offload_backend()` selects the appropriate backend based on
|
Factory function `get_offload_backend()` selects the appropriate backend based on
|
||||||
@@ -380,10 +369,4 @@ reasoner/generator components inside the model forward pass.
|
|||||||
**Notes:**
|
**Notes:**
|
||||||
- Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
|
- Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
|
||||||
- Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
|
- Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
|
||||||
- Distributed Layerwise Offloading reuses the layerwise block topology, but the
|
- Distributed Layerwise Offloading works with any model that supports Layerwise Offloading — no additional model changes required. See [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md) for usage examples.
|
||||||
model and parallelism combination still needs validation. AllGather mode
|
|
||||||
additionally requires a compatible checkpoint/mmap layout; no-AllGather mode
|
|
||||||
retains the standard loader's rank-local tensors. See the
|
|
||||||
[DLO design](../../design/feature/distributed_layerwise_offload.md) and
|
|
||||||
[Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
|
|
||||||
for usage examples.
|
|
||||||
@@ -56,14 +56,10 @@ vllm serve /path/to/Cosmos3-Super \
|
|||||||
--data-parallel-size 4
|
--data-parallel-size 4
|
||||||
```
|
```
|
||||||
|
|
||||||
### Without DLO AllGather (rank-local weights, no DLO sharding)
|
### Without AllGather (full weights per rank, no sharding)
|
||||||
|
|
||||||
Use `--dlo-no-use-allgather` when you want each rank to stream the standard
|
Use `--dlo-no-use-allgather` when you want each rank to load full weights independently —
|
||||||
loader's rank-local weights independently — no DLO AllGather synchronization,
|
no AllGather synchronization, no concurrent request requirement, but N× host memory.
|
||||||
no concurrent request requirement, but no DLO weight sharding and higher host
|
|
||||||
memory use. In pure DP, this is equivalent to independent full-weight
|
|
||||||
layerwise-offload replicas; with TP or HSDP, each rank retains the standard
|
|
||||||
loader's local tensor layout.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
vllm serve /path/to/Cosmos3-Nano \
|
vllm serve /path/to/Cosmos3-Nano \
|
||||||
@@ -158,9 +154,9 @@ wait
|
|||||||
| Flag | Description | Default |
|
| Flag | Description | Default |
|
||||||
|------|-------------|---------|
|
|------|-------------|---------|
|
||||||
| `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` |
|
| `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` |
|
||||||
| `--data-parallel-size N` | Number of DP ranks; DLO weight sharding + concurrent requests when AllGather is enabled | `1` |
|
| `--data-parallel-size N` | Number of DP ranks (weight sharding + concurrent requests) | `1` |
|
||||||
| `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` |
|
| `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` |
|
||||||
| `--dlo-no-use-allgather` | Stream standard-loader rank-local weights independently (no DLO sharding, no DLO AllGather) | `false` |
|
| `--dlo-no-use-allgather` | Each rank loads full weights independently (no sharding, no AllGather) | `false` |
|
||||||
| `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` |
|
| `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` |
|
||||||
|
|
||||||
## Mode comparison
|
## Mode comparison
|
||||||
@@ -170,7 +166,7 @@ wait
|
|||||||
| DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences |
|
| DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences |
|
||||||
| DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory |
|
| DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory |
|
||||||
| DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) |
|
| DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) |
|
||||||
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Standard-loader rank-local weights | 2 blocks | 0.4× HSDP | No DLO AllGather overhead, high CPU |
|
| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Full model | 2 blocks | 0.4× HSDP | No AllGather overhead, high CPU |
|
||||||
| HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM |
|
| HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM |
|
||||||
|
|
||||||
## Memory expectations
|
## Memory expectations
|
||||||
@@ -189,11 +185,7 @@ wait
|
|||||||
|
|
||||||
- `--dlo-use-allgather` (default) requires all concurrent requests to have the same
|
- `--dlo-use-allgather` (default) requires all concurrent requests to have the same
|
||||||
`num_inference_steps` (AllGather is a collective that needs all ranks synchronized)
|
`num_inference_steps` (AllGather is a collective that needs all ranks synchronized)
|
||||||
- Online quantization (FP8) is incompatible with the DLO+AllGather mmap path;
|
- Online quantization (FP8) is incompatible with mmap loading — falls back to regular
|
||||||
use `--dlo-no-use-allgather` or disable online quantization
|
`load_weights()` automatically
|
||||||
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path. TP with
|
- Tensor Parallel is not supported (DLO uses DP-based sharding, a different dimension)
|
||||||
`--dlo-no-use-allgather` retains the standard TP loader and should be treated
|
- HSDP + AllGather is rejected (would double-shard weights)
|
||||||
as an experimental compatibility path
|
|
||||||
- HSDP + AllGather is rejected (would double-shard weights). HSDP with
|
|
||||||
`--dlo-no-use-allgather` is accepted at configuration level, but is not yet a
|
|
||||||
broadly validated production combination
|
|
||||||
@@ -2,9 +2,6 @@
|
|||||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
"""Unit tests for LayerNorm and RMSNorm custom ops in diffusion layers."""
|
"""Unit tests for LayerNorm and RMSNorm custom ops in diffusion layers."""
|
||||||
|
|
||||||
import sys
|
|
||||||
import types
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
import torch
|
import torch
|
||||||
from pytest_mock import MockerFixture
|
from pytest_mock import MockerFixture
|
||||||
@@ -302,44 +299,6 @@ def test_rmsnorm_numerical_correctness():
|
|||||||
torch.testing.assert_close(out, expected, atol=1e-5, rtol=1e-5)
|
torch.testing.assert_close(out, expected, atol=1e-5, rtol=1e-5)
|
||||||
|
|
||||||
|
|
||||||
def test_rmsnorm_dtype_matches_parameter_and_fp32_accumulation_contract():
|
|
||||||
"""dtype controls gamma storage; reduced-precision inputs still use fp32 accumulation."""
|
|
||||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
|
||||||
|
|
||||||
hidden_size = 64
|
|
||||||
eps = 1e-6
|
|
||||||
norm = RMSNorm(hidden_size, eps=eps, dtype=torch.bfloat16)
|
|
||||||
x = torch.randn(2, 4, hidden_size, dtype=torch.bfloat16)
|
|
||||||
|
|
||||||
assert norm.weight.dtype == torch.bfloat16
|
|
||||||
|
|
||||||
x_fp32 = x.float()
|
|
||||||
expected = (x_fp32 * torch.rsqrt(x_fp32.square().mean(-1, keepdim=True) + eps) * norm.weight.float()).to(x.dtype)
|
|
||||||
|
|
||||||
torch.testing.assert_close(norm.forward_native(x), expected, atol=0, rtol=0)
|
|
||||||
|
|
||||||
|
|
||||||
def test_rmsnorm_npu_receives_gamma_with_configured_dtype(monkeypatch: pytest.MonkeyPatch):
|
|
||||||
"""The NPU fused op receives the bf16 checkpoint gamma used by H3."""
|
|
||||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
|
||||||
|
|
||||||
captured: dict[str, torch.Tensor] = {}
|
|
||||||
|
|
||||||
def npu_rms_norm(x: torch.Tensor, gamma: torch.Tensor, epsilon: float):
|
|
||||||
captured["gamma"] = gamma
|
|
||||||
captured["epsilon"] = torch.tensor(epsilon)
|
|
||||||
return (x,)
|
|
||||||
|
|
||||||
monkeypatch.setitem(sys.modules, "torch_npu", types.SimpleNamespace(npu_rms_norm=npu_rms_norm))
|
|
||||||
norm = RMSNorm(64, eps=1e-5, dtype=torch.bfloat16)
|
|
||||||
x = torch.randn(2, 4, 64, dtype=torch.bfloat16)
|
|
||||||
|
|
||||||
assert norm.forward_npu(x) is x
|
|
||||||
assert captured["gamma"] is norm.weight
|
|
||||||
assert captured["gamma"].dtype == torch.bfloat16
|
|
||||||
assert captured["epsilon"].item() == pytest.approx(1e-5)
|
|
||||||
|
|
||||||
|
|
||||||
# ── RMSNorm compile-path regression tests ──
|
# ── RMSNorm compile-path regression tests ──
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -10,13 +10,10 @@ forward_hip, and apply_rotary_emb_mindiesd paths.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import sys
|
|
||||||
import types
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
import torch
|
import torch
|
||||||
|
|
||||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding, apply_rotary_emb_mindiesd
|
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
||||||
|
|
||||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
@@ -237,33 +234,3 @@ class TestRotaryEmbeddingNativeShapeRegression:
|
|||||||
output = rope.forward_native(x, cos, sin)
|
output = rope.forward_native(x, cos, sin)
|
||||||
assert output.shape == x.shape
|
assert output.shape == x.shape
|
||||||
assert not torch.isnan(output).any()
|
assert not torch.isnan(output).any()
|
||||||
|
|
||||||
|
|
||||||
def test_mindie_full_dim_rope_keeps_h3_cos_sin_layout(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
||||||
"""H3 passes complete rotary dims to MindIE instead of the CUDA half-dim layout."""
|
|
||||||
captured: dict[str, object] = {}
|
|
||||||
|
|
||||||
def rotary_position_embedding(x, cos, sin, **kwargs):
|
|
||||||
captured.update(x=x, cos=cos, sin=sin, kwargs=kwargs)
|
|
||||||
return x
|
|
||||||
|
|
||||||
monkeypatch.setitem(
|
|
||||||
sys.modules,
|
|
||||||
"mindiesd",
|
|
||||||
types.SimpleNamespace(rotary_position_embedding=rotary_position_embedding),
|
|
||||||
)
|
|
||||||
x = torch.randn(11, 3, 96, dtype=torch.bfloat16)
|
|
||||||
cos = torch.randn(11, 96, dtype=torch.bfloat16)
|
|
||||||
sin = torch.randn(11, 96, dtype=torch.bfloat16)
|
|
||||||
|
|
||||||
actual = apply_rotary_emb_mindiesd(x, cos, sin, interleaved=False, half_head_dim=False)
|
|
||||||
|
|
||||||
assert actual is x
|
|
||||||
assert captured["x"] is x
|
|
||||||
assert captured["cos"] is cos
|
|
||||||
assert captured["sin"] is sin
|
|
||||||
assert captured["kwargs"] == {
|
|
||||||
"rotated_mode": "rotated_half",
|
|
||||||
"head_first": False,
|
|
||||||
"fused": True,
|
|
||||||
}
|
|
||||||
@@ -61,33 +61,6 @@ def test_packed_attention_is_a_regional_compile_boundary():
|
|||||||
assert getattr(MiniMaxH3Attention._run_packed_attention, "_torchdynamo_disable", False)
|
assert getattr(MiniMaxH3Attention._run_packed_attention, "_torchdynamo_disable", False)
|
||||||
|
|
||||||
|
|
||||||
def test_h3_fused_rope_matches_reference_and_preserves_unrotated_dims():
|
|
||||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
|
||||||
from vllm_omni.diffusion.models.minimax_h3.minimax_h3_transformer import (
|
|
||||||
MiniMaxH3Attention,
|
|
||||||
)
|
|
||||||
|
|
||||||
attention = object.__new__(MiniMaxH3Attention)
|
|
||||||
nn.Module.__init__(attention)
|
|
||||||
attention.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
|
|
||||||
attention.rope._forward_method = attention.rope.forward_native
|
|
||||||
|
|
||||||
x = torch.randn(11, 3, 128, dtype=torch.bfloat16)
|
|
||||||
freqs_half = torch.randn(11, 48)
|
|
||||||
freqs = torch.cat((freqs_half, freqs_half), dim=-1)
|
|
||||||
actual = attention._apply_rope(x, freqs)
|
|
||||||
|
|
||||||
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1)
|
|
||||||
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
|
|
||||||
x_rot = x[..., :96]
|
|
||||||
x1, x2 = x_rot.chunk(2, dim=-1)
|
|
||||||
expected_rot = x_rot * cos + torch.cat((-x2, x1), dim=-1) * sin
|
|
||||||
expected = torch.cat((expected_rot, x[..., 96:]), dim=-1)
|
|
||||||
|
|
||||||
torch.testing.assert_close(actual, expected, atol=0, rtol=0)
|
|
||||||
torch.testing.assert_close(actual[..., 96:], x[..., 96:], atol=0, rtol=0)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
("tp_size", "message"),
|
("tp_size", "message"),
|
||||||
[
|
[
|
||||||
|
|||||||
@@ -135,7 +135,6 @@ def _generate_joint_output(omni, config: QualityTestConfig):
|
|||||||
num_frames=config.num_frames,
|
num_frames=config.num_frames,
|
||||||
guidance_scale=config.guidance_scale,
|
guidance_scale=config.guidance_scale,
|
||||||
sigmas=config.sigmas,
|
sigmas=config.sigmas,
|
||||||
extra_args={"aspect_ratio": "16:9"},
|
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
main_process_peak_mem = torch.accelerator.max_memory_allocated() / (1024**3)
|
main_process_peak_mem = torch.accelerator.max_memory_allocated() / (1024**3)
|
||||||
|
|||||||
@@ -65,9 +65,9 @@ class LayerNorm(nn.LayerNorm, CustomOp):
|
|||||||
|
|
||||||
|
|
||||||
class RMSNorm(CustomOp):
|
class RMSNorm(CustomOp):
|
||||||
def __init__(self, hidden_size: int, eps: float = 1e-6, dtype: torch.dtype = torch.float32) -> None:
|
def __init__(self, hidden_size: int, eps: float = 1e-6) -> None:
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.weight = nn.Parameter(torch.ones(hidden_size, dtype=dtype))
|
self.weight = nn.Parameter(torch.ones(hidden_size))
|
||||||
self.variance_epsilon = eps
|
self.variance_epsilon = eps
|
||||||
|
|
||||||
def _forward_fused(self, x: torch.Tensor) -> torch.Tensor:
|
def _forward_fused(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
|||||||
@@ -89,11 +89,10 @@ class RotaryEmbedding(CustomOp):
|
|||||||
of 1st half and 2nd half (GPT-NeoX style).
|
of 1st half and 2nd half (GPT-NeoX style).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, is_neox_style: bool = False, half_head_dim: bool = True) -> None:
|
def __init__(self, is_neox_style: bool = False) -> None:
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.is_neox_style = is_neox_style
|
self.is_neox_style = is_neox_style
|
||||||
self.interleaved = not is_neox_style
|
self.interleaved = not is_neox_style
|
||||||
self.half_head_dim = half_head_dim
|
|
||||||
self.apply_rotary_emb_flash_attn = None
|
self.apply_rotary_emb_flash_attn = None
|
||||||
self.has_mindie = False
|
self.has_mindie = False
|
||||||
# ``find_spec("flash_attn")`` is True as long as *any* package publishes
|
# ``find_spec("flash_attn")`` is True as long as *any* package publishes
|
||||||
@@ -111,21 +110,6 @@ class RotaryEmbedding(CustomOp):
|
|||||||
if find_spec("mindiesd") is not None:
|
if find_spec("mindiesd") is not None:
|
||||||
self.has_mindie = True
|
self.has_mindie = True
|
||||||
|
|
||||||
def _prepare_half_head_dim_cos_sin(
|
|
||||||
self,
|
|
||||||
cos: torch.Tensor,
|
|
||||||
sin: torch.Tensor,
|
|
||||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
|
||||||
"""Normalize shared cos/sin to the half-dim layout used by CUDA/native kernels."""
|
|
||||||
if cos.dim() == 3:
|
|
||||||
# All batch elements share the same rotary position encoding.
|
|
||||||
cos = cos[0]
|
|
||||||
sin = sin[0]
|
|
||||||
if not self.half_head_dim:
|
|
||||||
cos = cos[..., ::2] if self.interleaved else cos[..., : cos.shape[-1] // 2]
|
|
||||||
sin = sin[..., ::2] if self.interleaved else sin[..., : sin.shape[-1] // 2]
|
|
||||||
return cos, sin
|
|
||||||
|
|
||||||
def forward_cuda(
|
def forward_cuda(
|
||||||
self,
|
self,
|
||||||
x: torch.Tensor,
|
x: torch.Tensor,
|
||||||
@@ -134,7 +118,10 @@ class RotaryEmbedding(CustomOp):
|
|||||||
) -> torch.Tensor:
|
) -> torch.Tensor:
|
||||||
from vllm.vllm_flash_attn.layers.rotary import apply_rotary_emb
|
from vllm.vllm_flash_attn.layers.rotary import apply_rotary_emb
|
||||||
|
|
||||||
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
|
if cos.dim() == 3:
|
||||||
|
# (B, S, D/2) -> (S, D/2)
|
||||||
|
cos = cos[0]
|
||||||
|
sin = sin[0]
|
||||||
|
|
||||||
x, squeezed = _ensure_batch_dim(x)
|
x, squeezed = _ensure_batch_dim(x)
|
||||||
output = apply_rotary_emb(
|
output = apply_rotary_emb(
|
||||||
@@ -154,7 +141,10 @@ class RotaryEmbedding(CustomOp):
|
|||||||
if self.apply_rotary_emb_flash_attn is None:
|
if self.apply_rotary_emb_flash_attn is None:
|
||||||
return self.forward_cuda(x, cos, sin)
|
return self.forward_cuda(x, cos, sin)
|
||||||
|
|
||||||
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
|
if cos.dim() == 3:
|
||||||
|
# (B, S, D/2) -> (S, D/2)
|
||||||
|
cos = cos[0]
|
||||||
|
sin = sin[0]
|
||||||
|
|
||||||
x, squeezed = _ensure_batch_dim(x)
|
x, squeezed = _ensure_batch_dim(x)
|
||||||
output = self.apply_rotary_emb_flash_attn(
|
output = self.apply_rotary_emb_flash_attn(
|
||||||
@@ -172,7 +162,7 @@ class RotaryEmbedding(CustomOp):
|
|||||||
sin: torch.Tensor,
|
sin: torch.Tensor,
|
||||||
) -> torch.Tensor:
|
) -> torch.Tensor:
|
||||||
if self.has_mindie:
|
if self.has_mindie:
|
||||||
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved, self.half_head_dim)
|
return apply_rotary_emb_mindiesd(x, cos, sin, self.interleaved)
|
||||||
else:
|
else:
|
||||||
return self.forward_native(x, cos, sin)
|
return self.forward_native(x, cos, sin)
|
||||||
|
|
||||||
@@ -201,7 +191,9 @@ class RotaryEmbedding(CustomOp):
|
|||||||
# All batch elements share the same rotary position encoding.
|
# All batch elements share the same rotary position encoding.
|
||||||
# Strip the batch dim so the underlying op broadcasts over the batch,
|
# Strip the batch dim so the underlying op broadcasts over the batch,
|
||||||
# consistent with forward_cuda / forward_hip / apply_rotary_emb_mindiesd.
|
# consistent with forward_cuda / forward_hip / apply_rotary_emb_mindiesd.
|
||||||
cos, sin = self._prepare_half_head_dim_cos_sin(cos, sin)
|
if cos.dim() == 3:
|
||||||
|
cos = cos[0]
|
||||||
|
sin = sin[0]
|
||||||
return apply_rotary_emb_torch(
|
return apply_rotary_emb_torch(
|
||||||
x,
|
x,
|
||||||
cos,
|
cos,
|
||||||
|
|||||||
@@ -33,8 +33,6 @@ from vllm_omni.diffusion.distributed.sp_plan import (
|
|||||||
SequenceParallelInput,
|
SequenceParallelInput,
|
||||||
SequenceParallelOutput,
|
SequenceParallelOutput,
|
||||||
)
|
)
|
||||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
|
||||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from vllm.model_executor.layers.quantization.base_config import (
|
from vllm.model_executor.layers.quantization.base_config import (
|
||||||
@@ -170,11 +168,16 @@ def _reorder_grouped_qkv_to_qkv(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> RMSNorm:
|
def _norm(size: int, *, eps: float, dtype: torch.dtype = _BF16_DTYPE) -> nn.RMSNorm:
|
||||||
# RMSNorm uses fp32 accumulation with bf16 inputs and outputs.
|
# RMSNorm uses fp32 accumulation with bf16 inputs and outputs.
|
||||||
# torch.nn.RMSNorm upcasts reduced-precision inputs for the variance
|
# torch.nn.RMSNorm upcasts reduced-precision inputs for the variance
|
||||||
# reduction, matching that accumulation semantic.
|
# reduction, matching that accumulation semantic.
|
||||||
return RMSNorm(size, eps=eps, dtype=dtype)
|
return nn.RMSNorm(size, eps=eps, dtype=dtype)
|
||||||
|
|
||||||
|
|
||||||
|
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
|
||||||
|
x1, x2 = torch.chunk(x, 2, dim=-1)
|
||||||
|
return torch.cat((-x2, x1), dim=-1)
|
||||||
|
|
||||||
|
|
||||||
def _modulate_scale_shift(
|
def _modulate_scale_shift(
|
||||||
@@ -227,6 +230,20 @@ class MiniMaxH3Rope(nn.Module):
|
|||||||
return torch.cat((half, half), dim=-1) # [S, 96]
|
return torch.cat((half, half), dim=-1) # [S, 96]
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_rope(x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
|
||||||
|
"""Rotate the first rot_dim head dims; pass the rest through.
|
||||||
|
|
||||||
|
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
|
||||||
|
are cast to the activation dtype before the elementwise math.
|
||||||
|
"""
|
||||||
|
rot_dim = freqs.shape[-1]
|
||||||
|
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
|
||||||
|
cos = torch.cos(freqs).to(x.dtype).unsqueeze(1) # [T, 1, rot_dim]
|
||||||
|
sin = torch.sin(freqs).to(x.dtype).unsqueeze(1)
|
||||||
|
x_rot = (x_rot * cos) + (_rotate_half(x_rot) * sin)
|
||||||
|
return torch.cat((x_rot, x_pass), dim=-1)
|
||||||
|
|
||||||
|
|
||||||
class MiniMaxH3TimeEmbedder(nn.Module):
|
class MiniMaxH3TimeEmbedder(nn.Module):
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -332,7 +349,6 @@ class MiniMaxH3Attention(nn.Module):
|
|||||||
self.num_kv_heads = self.qkv_proj.num_kv_heads
|
self.num_kv_heads = self.qkv_proj.num_kv_heads
|
||||||
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
||||||
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
||||||
self.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
|
|
||||||
self.out_proj = RowParallelLinear(
|
self.out_proj = RowParallelLinear(
|
||||||
inner_dim,
|
inner_dim,
|
||||||
arch.hidden_size,
|
arch.hidden_size,
|
||||||
@@ -351,19 +367,6 @@ class MiniMaxH3Attention(nn.Module):
|
|||||||
skip_sequence_parallel=skip_sequence_parallel,
|
skip_sequence_parallel=skip_sequence_parallel,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _apply_rope(self, x: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
|
|
||||||
"""Rotate the first rot_dim head dims; pass the rest through.
|
|
||||||
|
|
||||||
x: [T, heads, head_dim]; freqs: [T, rot_dim]. In the unfused path, cos/sin
|
|
||||||
are cast to the activation dtype before the elementwise math.
|
|
||||||
"""
|
|
||||||
rot_dim = freqs.shape[-1]
|
|
||||||
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
|
|
||||||
cos = torch.cos(freqs).to(x.dtype) # [T, rot_dim]
|
|
||||||
sin = torch.sin(freqs).to(x.dtype)
|
|
||||||
x_rot = self.rope(x_rot, cos, sin)
|
|
||||||
return torch.cat((x_rot, x_pass), dim=-1)
|
|
||||||
|
|
||||||
@torch.compiler.disable
|
@torch.compiler.disable
|
||||||
def _run_packed_attention(
|
def _run_packed_attention(
|
||||||
self,
|
self,
|
||||||
@@ -447,8 +450,8 @@ class MiniMaxH3Attention(nn.Module):
|
|||||||
q = self.q_norm(q)
|
q = self.q_norm(q)
|
||||||
k = self.k_norm(k)
|
k = self.k_norm(k)
|
||||||
if rope_freqs is not None:
|
if rope_freqs is not None:
|
||||||
q = self._apply_rope(q, rope_freqs)
|
q = _apply_rope(q, rope_freqs)
|
||||||
k = self._apply_rope(k, rope_freqs)
|
k = _apply_rope(k, rope_freqs)
|
||||||
|
|
||||||
# The packed layout uses a second document for alignment padding.
|
# The packed layout uses a second document for alignment padding.
|
||||||
# Local/Ulysses backends unpad it, while Ring keeps aligned rows for
|
# Local/Ulysses backends unpad it, while Ring keeps aligned rows for
|
||||||
|
|||||||
@@ -193,7 +193,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
|
|||||||
self.to(torch.float32)
|
self.to(torch.float32)
|
||||||
devices = [parameter.device] if parameter.device.type != "cpu" else []
|
devices = [parameter.device] if parameter.device.type != "cpu" else []
|
||||||
try:
|
try:
|
||||||
with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
|
with torch.random.fork_rng(devices=devices):
|
||||||
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
|
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
|
||||||
for device in devices:
|
for device in devices:
|
||||||
with self.device_module.device(device):
|
with self.device_module.device(device):
|
||||||
@@ -237,7 +237,7 @@ class MiniMaxH3VideoVAE(nn.Module, DistributedVaeMixin):
|
|||||||
self.to(torch.float32)
|
self.to(torch.float32)
|
||||||
devices = [parameter.device] if parameter.device.type != "cpu" else []
|
devices = [parameter.device] if parameter.device.type != "cpu" else []
|
||||||
try:
|
try:
|
||||||
with torch.random.fork_rng(devices=devices, device_type=parameter.device.type):
|
with torch.random.fork_rng(devices=devices):
|
||||||
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
|
torch.default_generator.manual_seed(MINIMAX_H3_KEYFRAME_ENCODE_SEED)
|
||||||
for device in devices:
|
for device in devices:
|
||||||
with self.device_module.device(device):
|
with self.device_module.device(device):
|
||||||
|
|||||||
Reference in new issue
Block a user