[Model] MammothModa2: enable Cache-DiT acceleration for the DiT stage (#7291)

Signed-off-by: shihongzhi <shi65881583@gmail.com>
This commit is contained in:
shihongzhi authored and GitHub committed 2026-09-21 18:34:12 -07:00
1 parent 0b5f8832b9
commit c11596c637
10 files changed
+507 -14

No files matched your search

+42 -6
View File
@@ -180,17 +180,52 @@ cache_dit.enable_cache(
> **Note:** For single transformer with multiple block lists, `refresh_context` works the same as standard models.
### Example 3: Sequential-CFG Diffusion Pipeline (MammothModa2)
MammothModa2's DiT stage runs on the diffusion runner, which enables the
configured backend at startup and transfers ownership to the pipeline through
the request-scoped protocol (`adopt_cache_dit_backend` / `is_cache_dit_enabled`);
`forward()` then reconciles per-request state through
`RequestScopedCacheDiTRuntime` (MiniMax H3 precedent).
**Key difference:** The enabler caches only the repeated main-layer stack and
marks the pipeline as requiring paired CFG forwards, because cache-dit's
separate-CFG accounting distinguishes cond/uncond passes purely by forward
parity:
```python
pipeline._cache_dit_requires_paired_cfg = True
block_adapter = BlockAdapter(
transformer=pipeline.gen_transformer,
blocks=[pipeline.gen_transformer.layers], # Refiners stay outside
forward_pattern=[ForwardPattern.Pattern_3],
has_separate_cfg=True,
check_forward_pattern=True,
)
```
Per request, the pipeline calls `runtime.prepare(spec)` for CFG requests
(refreshing the step context) and `runtime.prepare(None)` for `guidance=1.0`
requests (disabling hooks — their single forward per step cannot be expressed
by the parity accounting).
### Registering Custom Implementations
After writing your custom enabler, register it in `CUSTOM_DIT_ENABLERS` in
After writing your custom enabler, register it in
`CUSTOM_DIT_ENABLERS` via `register_custom_dit_enablers()` in
`vllm_omni/diffusion/cache/cachedit/model_specific.py`:
```python
CUSTOM_DIT_ENABLERS = {
"Wan22Pipeline": enable_cache_for_wan22,
"LongCatImagePipeline": enable_cache_for_longcat_image,
"YourCustomPipeline": enable_cache_for_your_model, # Add here
}
CUSTOM_DIT_ENABLERS.update(
{
"Wan22Pipeline": enable_cache_for_wan22,
"Cosmos3OmniDiffusersPipeline": enable_cache_for_cosmos3,
"Krea2Pipeline": enable_cache_for_krea2,
"Magi2Pipeline": enable_cache_for_magi2,
"MammothModa2DiTPipeline": enable_cache_for_mammothmoda2,
"YourCustomPipeline": enable_cache_for_your_model, # Add here
}
)
```
---
@@ -274,6 +309,7 @@ Complete examples in the codebase:
| **Wan2.2** | `cachedit.model_specific::enable_cache_for_wan22` | Single or dual-transformer | Auto-detects mode based on transformer_2 presence |
| **LongCat** | `cachedit.config::CacheDiTAdapterConfig` | Declarative block adapter | Two block lists in one transformer |
| **BAGEL** | `cachedit.model_specific::BagelCachedAdapter` | Custom cached adapter | Complex architecture |
| **MammothModa2** | `cachedit.model_specific::enable_cache_for_mammothmoda2` | Sub-stack + paired CFG | Generation-runner pipeline; lifecycle owned by `pipeline_mammothmoda2_dit.py` |
---
@@ -72,6 +72,39 @@ omni = Omni(
## Example Script
### Multi-Stage Models (MammothModa2)
MammothModa2 is a multi-stage AR→DiT pipeline; the diffusion runner owns the
Cache-DiT lifecycle of its DiT stage. Configure it on the **DiT stage entry**
of the deploy YAML so the acceleration only applies to the denoising stage:
```yaml
# deploy YAML (see vllm_omni/deploy/mammoth_moda2.yaml)
stages:
- stage_id: 1
# ... other DiT stage settings ...
cache_backend: cache_dit
cache_config:
Fn_compute_blocks: 1
Bn_compute_blocks: 0
max_warmup_steps: 4
residual_diff_threshold: 0.24
max_continuous_cached_steps: 3
enable_cache_dit_summary: true # log skip-ratio stats per request
```
Behavior notes:
- Only the repeated main-layer stack is cached; the Q-Former refiners always run.
- Sequential-CFG parity requires the unconditional pass on every denoise step,
so the `cfg_range` skip optimization is not used: outside the interval CFG is
neutralized with `scale=1.0` instead of skipping the uncond forward.
- Requests with `text_guidance_scale = 1.0` run with cache hooks disabled.
- `residual_diff_threshold` defaults to `0.24` (measured ~1.7-2.2x at 50 steps,
PSNR 27-34 dB vs uncached). For 20-step generation — especially with a
partial `cfg_range` — use `0.12` for noticeably closer parity at a modest
speed cost.
### Offline Inference
Use the example script under `examples/offline_inference/text_to_image`:
+2 -1
View File
@@ -130,7 +130,7 @@ The following tables show which models support each feature:
| **Krea 2** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LongCat-Image** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **LongCat-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **MammothModa2(T2I)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **MammothModa2(T2I)** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Nextstep_1(T2I)** | ❓ | ❓ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **OmniGen2** | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Ovis-Image** | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
@@ -154,6 +154,7 @@ The following tables show which models support each feature:
> 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`.
> 5. HunyuanImage3 supports step execution. Multi-request step execution requires `TORCH_SDPA`; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution).
> 6. BAGEL step execution supports image generation with `bagel.yaml`, `bagel_think.yaml`, and `bagel_single_stage.yaml`; two-stage Thinker execution and explicit single-stage text output remain on their existing complete-request paths. Image requests require `num_inference_steps >= 2`. BAGEL step execution cannot currently be combined with sequence parallelism or a diffusion cache backend; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution).
> 7. MammothModa2 runs its DiT stage on the diffusion runner (`StageExecutionType.DIFFUSION`); Cache-DiT is enabled through the standard diffusion-stage knobs on the stage entry of the deploy YAML (`cache_backend: cache_dit`, plus optional `cache_config` / `enable_cache_dit_summary`). The runner installs the backend at startup and the pipeline adopts it per request. Only the repeated main-layer stack is cached; requests with `text_guidance_scale = 1.0` bypass the cache hooks.
### VideoGen
@@ -157,7 +157,11 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--enable-cache-dit-summary",
action="store_true",
help="Enable cache-dit summary logging after diffusion forward passes.",
default=None,
help=(
"Enable cache-dit summary logging after diffusion forward passes. "
"Default: unset (defer to the deploy YAML's enable_cache_dit_summary)."
),
)
parser.add_argument(
"--ulysses-degree",
+108
View File
@@ -60,6 +60,7 @@ def test_custom_cache_dit_enablers_are_registered_explicitly():
"Cosmos3OmniPipeline": cd_model_specific.enable_cache_for_cosmos3,
"Krea2Pipeline": cd_model_specific.enable_cache_for_krea2,
"Magi2Pipeline": cd_model_specific.enable_cache_for_magi2,
"MammothModa2DiTPipeline": cd_model_specific.enable_cache_for_mammothmoda2,
}
with patch.dict(cd_backend.CUSTOM_DIT_ENABLERS, {}, clear=True):
@@ -116,6 +117,33 @@ def test_magi2_cache_dit_targets_only_nested_repeated_layers(mock_block_adapter,
assert get_transformer(pipeline) is transformer_block
@patch("vllm_omni.diffusion.cache.cachedit.model_specific.enable_cache_for_dit")
@patch("vllm_omni.diffusion.cache.cachedit.model_specific.BlockAdapter")
def test_mammothmoda2_cache_dit_targets_only_main_layers(mock_block_adapter, mock_enable_cache):
pipeline = Mock()
transformer = pipeline.gen_transformer
layers = torch.nn.ModuleList([torch.nn.Identity()])
transformer.layers = layers
adapter = mock_block_adapter.return_value
refresh = Mock()
mock_enable_cache.return_value = refresh
result = cd_model_specific.enable_cache_for_mammothmoda2(pipeline, SAMPLE_CACHE_CONFIG)
mock_block_adapter.assert_called_once()
adapter_kwargs = mock_block_adapter.call_args.kwargs
assert adapter_kwargs["transformer"] is transformer
assert adapter_kwargs["blocks"] == [layers]
assert adapter_kwargs["has_separate_cfg"] is True
assert adapter_kwargs["check_forward_pattern"] is True
assert result.refresh is refresh
assert result.targets == (adapter,)
get_transformer = mock_enable_cache.call_args.kwargs["get_pipeline_transformer"]
assert get_transformer(pipeline) is transformer
# Sequential-CFG parity is mandatory while cache-dit is installed.
assert pipeline._cache_dit_requires_paired_cfg is True
@patch("vllm_omni.diffusion.cache.cachedit.backend.cache_dit.summary")
@patch("vllm_omni.diffusion.cache.cachedit.backend.BlockAdapter.is_cached", return_value=True)
def test_cache_summary_uses_custom_nested_targets(mock_is_cached, mock_summary):
@@ -338,3 +366,83 @@ def test_summary_with_no_transformer_is_nonfatal():
pass
cache_summary(pipeline=FakePipeline())
# This test is skipped on ROCm since rocm_unquantized_gemm doesn't support CPU backend
@pytest.mark.skipif(
current_omni_platform.is_rocm(),
reason="vLLM ROCm custom ops lack CPU fallback",
)
def test_mammothmoda2_cache_dit_runs_end_to_end_on_tiny_model(request: pytest.FixtureRequest):
"""A tiny MammothModa2 DiT runs through Cache-DiT with paired CFG forwards.
Exercises the Pattern_3 contract end to end: after enabling the custom
enabler, blocks must be callable as ``block(hidden_states, **kwargs)``
and the pipeline's two-forwards-per-step CFG cadence must drive the
cache context without shape or parity errors.
"""
from cache_dit import BlockAdapter
from vllm_omni.diffusion.models.mammoth_moda2.mammothmoda2_dit_model import Transformer2DModel
from vllm_omni.diffusion.models.mammoth_moda2.rope_real import RotaryPosEmbedReal
# MammothModa2 attention resolves through the shared Omni attention layer,
# which picks the platform-default backend at construction time. Pin
# TORCH_SDPA so a CUDA-visible host does not select FA3 for CPU tensors.
with _force_torch_sdpa():
model = Transformer2DModel(
patch_size=2,
in_channels=4,
hidden_size=96,
num_layers=4,
num_refiner_layers=1,
num_attention_heads=2,
num_kv_heads=2,
multiple_of=8,
axes_dim_rope=(16, 16, 16),
axes_lens=(300, 512, 512),
text_feat_dim=16,
)
model.eval()
MammothModa2Pipeline = type("MammothModa2DiTPipeline", (), {})
pipeline = MammothModa2Pipeline()
pipeline.gen_transformer = model
backend = CacheDiTBackend(DiffusionCacheConfig())
backend.enable(pipeline)
request.addfinalizer(lambda: backend.disable(pipeline))
assert BlockAdapter.is_cached(model)
num_inference_steps = 8
backend.refresh(pipeline, num_inference_steps=num_inference_steps)
freqs_cis = RotaryPosEmbedReal.get_freqs_real((16, 16, 16), (300, 512, 512), theta=10000)
latents = torch.randn(1, 4, 8, 8)
text = torch.randn(1, 5, 16)
negative = torch.randn(1, 3, 16)
text_mask = torch.ones(1, 5, dtype=torch.bool)
negative_mask = torch.ones(1, 3, dtype=torch.bool)
with torch.no_grad():
for step in range(num_inference_steps):
timestep = torch.full((1,), 1.0 - step / num_inference_steps)
# Conditional pass ...
cond = model(
hidden_states=latents,
timestep=timestep,
text_hidden_states=text,
freqs_cis=freqs_cis,
text_attention_mask=text_mask,
)
# ... always followed by the unconditional pass (parity).
uncond = model(
hidden_states=latents,
timestep=timestep,
text_hidden_states=negative,
freqs_cis=freqs_cis,
text_attention_mask=negative_mask,
)
latents = uncond + 4.0 * (cond - uncond)
assert latents.shape == (1, 4, 8, 8)
assert torch.isfinite(latents).all()
@@ -1,13 +1,27 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from contextlib import contextmanager
from dataclasses import dataclass
from types import SimpleNamespace
import cache_dit
import pytest
import torch
from cache_dit import BlockAdapter
from torch import nn
from vllm_omni.diffusion.data import DiffusionOutput, OmniDiffusionConfig, TransformerConfig
from vllm_omni.diffusion.cache.cachedit import CacheDiTBackend, RequestScopedCacheDiTRuntime
from vllm_omni.diffusion.cache.selector import get_cache_backend
from vllm_omni.diffusion.config import set_current_diffusion_config
from vllm_omni.diffusion.data import (
AttentionConfig,
DiffusionCacheConfig,
DiffusionOutput,
OmniDiffusionConfig,
TransformerConfig,
)
from vllm_omni.diffusion.models.interface import adopt_request_scoped_cache_dit
from vllm_omni.diffusion.models.mammoth_moda2 import pipeline_mammothmoda2_dit
from vllm_omni.diffusion.models.mammoth_moda2.pipeline_mammothmoda2_dit import (
MammothModa2DiTPipeline,
@@ -17,6 +31,7 @@ from vllm_omni.diffusion.models.mammoth_moda2.pipeline_mammothmoda2_dit import (
from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.diffusion.worker.request_batch import DiffusionRequestBatch
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.platforms import current_omni_platform
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -110,6 +125,10 @@ def _pipeline_shell() -> MammothModa2DiTPipeline:
pipeline.device = torch.device("cpu")
pipeline.config = _build_mammoth_config(_od_config())
pipeline._llm_hidden_size = 8
# Cache-DiT runtime state that __init__ normally creates (disabled here,
# matching a pipeline built without a cache_backend).
pipeline._cache_dit_runtime = RequestScopedCacheDiTRuntime(pipeline)
pipeline._cache_dit_config = None
return pipeline
@@ -118,12 +137,14 @@ def _batch(
request_id: str = "req-7",
prompt: object | None = None,
sampling: OmniDiffusionSamplingParams | None = None,
height: int = 32,
width: int = 48,
) -> DiffusionRequestBatch:
if prompt is None:
prompt = {
"prompt": "",
"height": 32,
"width": 48,
"height": height,
"width": width,
"additional_information": {
"full_hidden_states": torch.arange(32, dtype=torch.float32).reshape(4, 8),
"full_token_ids": [10, 11, 100, 101],
@@ -390,3 +411,162 @@ def test_forward_rejects_missing_visual_tokens_before_model_access() -> None:
}
with pytest.raises(ValueError, match="no visual-token hidden states.*req-empty"):
_pipeline_shell().forward(_batch(request_id="req-empty", prompt=prompt))
@contextmanager
def _force_torch_sdpa():
"""Pin TORCH_SDPA so CPU shape tests do not pick CUDA-only backends (FA3)."""
od_config = SimpleNamespace(
diffusion_attention_config=AttentionConfig(default="TORCH_SDPA"),
parallel_config=SimpleNamespace(ring_degree=1),
)
with set_current_diffusion_config(od_config):
yield
def _constructible_raw_config() -> dict:
"""Checkpoint config whose DiT/VAE are small enough to actually run on CPU."""
return {
"model_type": "mammothmoda2",
"llm_config": {
"model_type": "mammothmoda2_qwen2_5_vl",
"text_config": {
"model_type": "mammothmoda2_qwen2_5_vl_text",
"hidden_size": 8,
"gen_vocab_start_index": 100,
},
},
"gen_vae_config": {"block_out_channels": [8, 8], "norm_num_groups": 8},
"gen_dit_config": {
"patch_size": 2,
"in_channels": 4,
"hidden_size": 96,
"num_layers": 4,
"num_refiner_layers": 1,
"num_attention_heads": 2,
"num_kv_heads": 2,
"multiple_of": 8,
"axes_dim_rope": [16, 16, 16],
"axes_lens": [300, 512, 512],
"text_feat_dim": 16,
},
# Top-level axes feed the pipeline-owned rotary table (see pipeline __init__).
"gen_axes_dim_rope": [16, 16, 16],
"gen_axes_lens": [300, 512, 512],
}
def _cache_dit_od_config() -> OmniDiffusionConfig:
return OmniDiffusionConfig(
model="/models/MammothModa2-Preview",
model_class_name="MammothModa2DiTPipeline",
tf_model_config=TransformerConfig.from_dict(_constructible_raw_config()),
cache_backend="cache_dit",
cache_config=DiffusionCacheConfig(),
)
@pytest.mark.skipif(
current_omni_platform.is_rocm(),
reason="vLLM ROCm custom ops lack CPU fallback",
)
def test_forward_transitions_request_scoped_cache_dit_across_requests(request: pytest.FixtureRequest) -> None:
"""Cache-DiT request lifecycle on one production-built pipeline instance.
Mirrors the diffusion runner startup (``get_cache_backend`` → ``enable`` →
``adopt_request_scoped_cache_dit``) on a tiny pipeline built through the
real ``__init__``, then drives three requests through the real ``forward()``:
1. a partial-CFG request installs the request-scoped cache context:
cond/uncond forwards stay paired on every denoise step (parity
accounting), even outside ``cfg_range``, and blocks record cached steps;
2. a guidance=1.0 request runs with hooks disabled (``prepare(None)``) and
its output is bit-identical to a reference captured before the backend
was ever enabled, proving a clean uninstall with no stale module state;
3. a CFG request with a different step count and image size re-installs the
hooks through a fresh context, so no stale shape or step count survives.
"""
torch.manual_seed(0) # Deterministic random init → deterministic cache hits.
with _force_torch_sdpa():
pipeline = MammothModa2DiTPipeline(od_config=_cache_dit_od_config())
pipeline.eval()
assert pipeline._cache_dit_config is not None
request.addfinalizer(pipeline._cache_dit_runtime.disable)
# The default prompt carries 2 text + 2 visual condition tokens, so the
# conditional pass sees 4 tokens and the uncond pass (empty prompt) sees 0.
cond_tokens, uncond_tokens = 4, 0
seq_lens: list[int] = []
def record_call(_module, _args, kwargs):
text = kwargs.get("text_hidden_states")
seq_lens.append(int(text.shape[1]) if text is not None else -1)
hook = pipeline.gen_transformer.register_forward_pre_hook(record_call, with_kwargs=True)
unguided_sampling = OmniDiffusionSamplingParams(
height=32, width=48, seed=4321, guidance_scale=1.0, num_inference_steps=8
)
partial_cfg_sampling = OmniDiffusionSamplingParams(
height=32,
width=48,
seed=1234,
guidance_scale=4.0,
num_inference_steps=8,
extra_args={"cfg_range": [0.2, 0.8]},
)
changed_cfg_sampling = OmniDiffusionSamplingParams(
height=48,
width=64,
seed=777,
guidance_scale=6.0,
num_inference_steps=10,
extra_args={"cfg_range": [0.1, 0.9]},
)
# Reference run for phase 2: identical request while hooks were never
# installed on this pipeline.
assert not pipeline.is_cache_dit_enabled()
reference = pipeline.forward(_batch(sampling=unguided_sampling)).output
assert seq_lens == [cond_tokens] * 8
assert torch.isfinite(reference.float()).all()
# Runner startup: enable once, then the pipeline owns all transitions.
backend = get_cache_backend("cache_dit", DiffusionCacheConfig())
assert isinstance(backend, CacheDiTBackend)
backend.enable(pipeline)
assert adopt_request_scoped_cache_dit(pipeline, backend)
assert pipeline.is_cache_dit_enabled()
# Phase 1: partial CFG keeps cond/uncond paired on every step, including
# the steps whose normalized index falls outside cfg_range [0.2, 0.8].
seq_lens.clear()
pipeline.forward(_batch(sampling=partial_cfg_sampling))
assert seq_lens == [cond_tokens, uncond_tokens] * 8
assert pipeline.is_cache_dit_enabled()
assert BlockAdapter.is_cached(pipeline.gen_transformer)
hits = cache_dit.summary(pipeline.gen_transformer, logging=False)[0]
assert hits.cached_steps or hits.cfg_cached_steps
# Phase 2: no-CFG request disables the hooks and must reproduce the
# never-enabled reference bit-for-bit.
seq_lens.clear()
unguided = pipeline.forward(_batch(sampling=unguided_sampling)).output
assert seq_lens == [cond_tokens] * 8
assert not pipeline.is_cache_dit_enabled()
assert not BlockAdapter.is_cached(pipeline.gen_transformer)
assert torch.equal(unguided, reference)
# Phase 3: CFG with a new step count and image size re-installs the hooks
# on a fresh context; output shape follows the new request geometry.
seq_lens.clear()
regenerated = pipeline.forward(_batch(sampling=changed_cfg_sampling, height=48, width=64)).output
assert seq_lens == [cond_tokens, uncond_tokens] * 10
assert pipeline.is_cache_dit_enabled()
assert BlockAdapter.is_cached(pipeline.gen_transformer)
hits = cache_dit.summary(pipeline.gen_transformer, logging=False)[0]
assert hits.cached_steps or hits.cfg_cached_steps
assert regenerated.shape == (1, 3, 12, 16)
assert torch.isfinite(regenerated.float()).all()
hook.remove()
+10
View File
@@ -20,3 +20,13 @@ stages:
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: false
# Optional Cache-DiT acceleration for the DiT stage (~2x faster denoise,
# training-free). Uncomment to enable; guidance=1.0 requests bypass it.
# cache_backend: cache_dit
# cache_config:
# Fn_compute_blocks: 1
# Bn_compute_blocks: 0
# max_warmup_steps: 4
# residual_diff_threshold: 0.24
# max_continuous_cached_steps: 3
# enable_cache_dit_summary: true
+42
View File
@@ -867,6 +867,47 @@ def enable_cache_for_magi2(pipeline: Any, cache_config: Any) -> CacheDiTEnableRe
return CacheDiTEnableResult(refresh=refresh, targets=(block_adapter,))
def _get_mammothmoda2_transformer(pipeline: Any) -> torch.nn.Module:
return pipeline.gen_transformer
def enable_cache_for_mammothmoda2(pipeline: Any, cache_config: Any) -> CacheDiTEnableResult:
"""Cache only MammothModa2's repeated main DiT stack.
``Transformer2DModel`` runs three Q-Former refiners (noise / ref-image /
context) whose inputs change every denoise step, plus the ``layers``
stack that dominates per-step compute. Only ``layers`` is a repeated
residual stack, so the ``BlockAdapter`` wraps it and the refiners stay
outside the cached region. Blocks take ``hidden_states`` plus keyword
step context and return only hidden states (``Pattern_3``).
The pipeline runs sequential CFG: each conditional forward is followed by
an unconditional forward. cache-dit tells cond/uncond apart purely by
transformer-forward parity (``has_separate_cfg=True``), so the
``cfg_range`` optimization that skips the unconditional pass outside the
interval would desync that accounting. Like Cosmos3, we keep the passes
paired and neutralize CFG via scale=1.0 outside the interval; the
pipeline also disables hooks for no-CFG requests (single forward per
step), whose parity the accounting cannot represent.
"""
pipeline._cache_dit_requires_paired_cfg = True
transformer = _get_mammothmoda2_transformer(pipeline)
block_adapter = BlockAdapter(
transformer=transformer,
blocks=[transformer.layers],
forward_pattern=[ForwardPattern.Pattern_3],
has_separate_cfg=True,
check_forward_pattern=True,
)
refresh = enable_cache_for_dit(
pipeline,
cache_config,
block_adapter,
get_pipeline_transformer=_get_mammothmoda2_transformer,
)
return CacheDiTEnableResult(refresh=refresh, targets=(block_adapter,))
def register_custom_dit_enablers() -> None:
"""Register model-specific Cache-DiT enablers.
@@ -885,6 +926,7 @@ def register_custom_dit_enablers() -> None:
"Cosmos3OmniPipeline": enable_cache_for_cosmos3,
"Krea2Pipeline": enable_cache_for_krea2,
"Magi2Pipeline": enable_cache_for_magi2,
"MammothModa2DiTPipeline": enable_cache_for_mammothmoda2,
}
)
@@ -695,8 +695,16 @@ class Transformer2DModel(ModelMixin, ConfigMixin):
return text_hidden_states, img_tokens
def _apply_transformer_layers(self, hidden_states, attention_mask, rotary_emb, temb):
# Call blocks with keyword arguments: Cache-DiT's Pattern_3 wrapper
# re-invokes blocks as block(hidden_states, **kwargs), so the cached
# and uncached paths share this exact calling convention.
for layer in self.layers:
hidden_states = layer(hidden_states, attention_mask, rotary_emb, temb)
hidden_states = layer(
hidden_states,
attention_mask=attention_mask,
image_rotary_emb=rotary_emb,
temb=temb,
)
return hidden_states
def forward(
@@ -15,7 +15,12 @@ from transformers.models.qwen2.modeling_qwen2 import Qwen2RMSNorm
from vllm.logger import init_logger
from vllm.model_executor.models.utils import AutoWeightsLoader, WeightsMapper
from vllm_omni.diffusion.data import DiffusionOutput, OmniDiffusionConfig
from vllm_omni.diffusion.cache.cachedit import (
CacheDiTBackend,
CacheDiTRequestSpec,
RequestScopedCacheDiTRuntime,
)
from vllm_omni.diffusion.data import DiffusionCacheConfig, DiffusionOutput, OmniDiffusionConfig
from vllm_omni.diffusion.distributed.utils import get_local_device
from vllm_omni.diffusion.model_loader.diffusers_loader import DiffusersPipelineLoader
from vllm_omni.diffusion.models.interface import SupportsComponentDiscovery
@@ -28,6 +33,9 @@ from .schedulers import FlowMatchEulerDiscreteScheduler
logger = init_logger(__name__)
# Identifies the pipeline-owned Cache-DiT installation across requests.
_MAMMOTHMODA2_CACHE_DIT_KEY = "mammothmoda2:cache_dit"
def _first_request_value(value: object) -> object:
if isinstance(value, (list, tuple)):
@@ -169,6 +177,54 @@ class MammothModa2DiTPipeline(nn.Module, SupportsComponentDiscovery):
self._llm_hidden_size = llm_hidden_size
# Cache-DiT lifecycle: the diffusion runner enables the configured
# backend (``cache_backend`` on the deploy YAML stage entry) at startup
# and transfers ownership here via the request-scoped protocol;
# forward() then reconciles per-request state (step count, CFG parity).
# The runner also emits the cache summary when
# ``enable_cache_dit_summary`` is set.
self._cache_dit_runtime = RequestScopedCacheDiTRuntime(self)
self._cache_dit_config: DiffusionCacheConfig | None = None
if str(getattr(od_config, "cache_backend", "") or "").lower() == "cache_dit":
cache_config = od_config.cache_config
self._cache_dit_config = (
cache_config
if isinstance(cache_config, DiffusionCacheConfig)
else DiffusionCacheConfig.from_dict(cache_config or {})
)
def adopt_cache_dit_backend(self, backend: CacheDiTBackend) -> None:
"""Adopt a runner-installed Cache-DiT backend (request-scoped protocol)."""
self._cache_dit_runtime.adopt(backend, installation_key=_MAMMOTHMODA2_CACHE_DIT_KEY)
def is_cache_dit_enabled(self) -> bool:
"""Return the request-scoped Cache-DiT installation state."""
return self._cache_dit_runtime.is_enabled
def _prepare_cache_dit_for_request(self, *, num_inference_steps: int, cfg_active: bool) -> None:
"""Reconcile Cache-DiT hooks with this request before the denoise loop.
cache-dit's separate-CFG accounting assumes exactly two transformer
forwards per denoise step (conditional then unconditional), so CFG
requests refresh the context (per-request step counts vary) while
no-CFG requests run with hooks disabled: their single forward per
step cannot be represented by the parity accounting.
"""
if self._cache_dit_config is None:
return
if not cfg_active:
self._cache_dit_runtime.prepare(None)
return
self._cache_dit_runtime.prepare(
CacheDiTRequestSpec(
installation_key=_MAMMOTHMODA2_CACHE_DIT_KEY,
cache_config=self._cache_dit_config,
num_inference_steps=num_inference_steps,
)
)
def _reinit_caption_embedder(self, in_features: int) -> None:
# Align with upstream Mammothmoda2Model's `reinit_caption_embedder`:
# Use Qwen2RMSNorm(in_features) + Linear(in_features -> out_features).
@@ -432,6 +488,18 @@ class MammothModa2DiTPipeline(nn.Module, SupportsComponentDiscovery):
# Run diffusion loop (CFG supported when text_guidance_scale > 1.0)
total_steps = max(1, len(scheduler.timesteps))
self._prepare_cache_dit_for_request(
num_inference_steps=total_steps,
cfg_active=request.text_guidance_scale > 1.0,
)
# cache-dit distinguishes the conditional vs unconditional passes
# purely by transformer-forward parity, so when it is active the
# uncond pass must run on every step. Outside cfg_range CFG is
# neutralized via scale=1.0 (uncond + 1.0 * (cond - uncond) == cond),
# matching the skip optimization's cond-only update.
requires_paired_cfg = self._cache_dit_runtime.is_enabled and getattr(
self, "_cache_dit_requires_paired_cfg", False
)
for i, t in enumerate(scheduler.timesteps):
timestep = t.expand(latents.shape[0]).to(latents.dtype)
model_pred = self.gen_transformer(
@@ -447,7 +515,10 @@ class MammothModa2DiTPipeline(nn.Module, SupportsComponentDiscovery):
guidance_scale = (
request.text_guidance_scale if request.cfg_range[0] <= i / total_steps <= request.cfg_range[1] else 1.0
)
if guidance_scale > 1.0 and negative_prompt_embeds is not None:
run_uncond = guidance_scale > 1.0 and negative_prompt_embeds is not None
if requires_paired_cfg and negative_prompt_embeds is not None:
run_uncond = True
if run_uncond:
model_pred_uncond = self.gen_transformer(
hidden_states=latents,
timestep=timestep,