Add normalized online FP8 cache for DLO

Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-19 11:55:29 +08:00
1 parent 14bd8f8772
commit 1ae56cfcfb
14 files changed
+714 -11

No files matched your search

@@ -105,6 +105,31 @@ flags such as `_supports_mmap_loading` or parameter attributes for mmap-only
transforms. Model-specific direct-layout knowledge, when required, lives in a
checkpoint adapter beside the ordinary loader.
### Phase-I normalized online-FP8 cache
The direct-checkpoint plan above cannot quantize a BF16 source online. Phase I
adds an explicit, opt-in normalized cache for the narrower case of one TP1
transformer with per-tensor online FP8 and DLO AllGather:
```text
cache miss: BF16 source -> ordinary online FP8 loader -> atomic FP8 cache
cache hit: FP8 cache -> direct mmap plan -> DLO host shard -> AllGather
```
The cache stores the serialized runtime representation, including finalized FP8
weights, generated scales, and persistent transformer buffers. Its manifest
records the schema, component, quantization contract, tensor count, source
files, and a source fingerprint. A missing, stale, incomplete, incompatible, or
invalid artifact is a cache miss; cache publication is best-effort and falls
back to the ordinary loader on failure.
Phase I keeps the cache binding deliberately simple: one source per transformer
component, with existing per-tensor transforms applied before publication. The
cache is not selected for DLO-disabled, no-AllGather, TP>1, or HSDP deployments,
and it normalizes the transformer only. Cache sharing and compatibility for
other components, transformed TP coordinates, and broader DP/SP layouts remain
in [RFC #6231](https://github.com/vllm-project/vllm-omni/issues/6231).
### AllGather path
With the default `dlo_use_allgather=True`, each rank stores approximately
@@ -211,10 +236,10 @@ reconstructs those tensors with their recorded layouts. Other online methods
must use `--dlo-no-use-allgather` or disable online quantization until their
runtime layouts are validated.
A normalized runtime mmap cache, built through the ordinary loader, is the
proposed general mechanism for sharing transformed TP or quantized layouts.
That cache and its publication/lifecycle protocol are intentionally outside
this phase; see [RFC #6195](https://github.com/vllm-project/vllm-omni/issues/6195).
The Phase-I normalized online-FP8 cache is the supported TP1 transformer case.
The general cache-compatibility contract for additional components, transformed
TP coordinates, and cross-DP/SP reuse remains in
[RFC #6231](https://github.com/vllm-project/vllm-omni/issues/6231).
## Validation coverage
@@ -42,6 +42,12 @@ vllm serve /path/to/model --omni \
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--usp 4
# Opt-in normalized online-FP8 cache (TP1 + DLO AllGather)
vllm serve /path/to/model --omni \
--quantization fp8 \
--enable-distributed-layerwise-offload \
--dlo-fp8-cache-dir /path/to/node-local/fp8-cache
```
```python
@@ -63,6 +69,7 @@ omni = Omni(
| `--dlo-use-allgather` | Shard host weights and reconstruct with AllGather | `true` |
| `--dlo-no-use-allgather` | Stream complete rank-local blocks without a DLO weight collective | `false` |
| `--dlo-resident-layers N` | Keep N leading main-DiT blocks on device; requires no-AllGather and model-declared resident paths | `0` |
| `--dlo-fp8-cache-dir PATH` | Opt into the Phase-I normalized online-FP8 cache for the transformer | unset |
## Host-weight loading
@@ -104,6 +111,24 @@ process.
When the effective DLO group size is one, `dlo_use_allgather=True` does not
perform a collective and uses the same rank-local transfer behavior.
## Normalized online-FP8 cache
`--dlo-fp8-cache-dir` is an opt-in Phase-I cache for per-tensor online FP8 with
TP1 and DLO AllGather. On a cache miss, the ordinary loader quantizes the
transformer once and publishes finalized FP8 weights and scales as an atomic,
serialized safetensors artifact. On a valid cache hit, the loader selects that
serialized source before DLO planning, so the transformer can be loaded through
the direct checkpoint-mmap path without re-materializing the full BF16 source.
The cache is keyed to the source files through the manifest's schema, component,
quantization contract, and source fingerprint. Incomplete, incompatible, or
stale artifacts are treated as cache misses. Cache creation is best-effort: a
publication failure logs a warning and keeps the ordinary online-FP8 path.
Phase I normalizes the transformer only. Other pipeline components keep their
ordinary loading and quantization behavior. The cache is not selected when DLO
is disabled, AllGather is disabled, TP is greater than one, or HSDP is enabled.
## Declarative topology
Models may declare an `OffloadPlan` instead of embedding offload logic:
@@ -147,10 +172,9 @@ must enter each collective.
`OffloadPlan` that declares eligible `resident_dit_paths`.
- DP concurrency requires an explicit, identical inference-step count.
Sharing transformed TP or quantized runtime layouts through a normalized mmap
cache is a follow-up design in
[RFC #6195](https://github.com/vllm-project/vllm-omni/issues/6195), not part of
the direct-checkpoint path.
Sharing transformed TP or quantized runtime layouts beyond this TP1 transformer
cache remains a follow-up design in
[RFC #6231](https://github.com/vllm-project/vllm-omni/issues/6231).
See the [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
for an end-to-end example.